Aug 04 18:08:42.537806 ip-10-0-136-165 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Aug 04 18:08:42.537819 ip-10-0-136-165 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Aug 04 18:08:42.537828 ip-10-0-136-165 systemd[1]: kubelet.service: Failed with result 'resources'. Aug 04 18:08:42.538123 ip-10-0-136-165 systemd[1]: Failed to start Kubernetes Kubelet. Aug 04 18:08:53.810463 ip-10-0-136-165 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Aug 04 18:08:53.810480 ip-10-0-136-165 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d8f79cd5c9734b4b8e89c3986aef9362 -- Aug 04 18:10:42.657214 ip-10-0-136-165 systemd[1]: Starting Kubernetes Kubelet... Aug 04 18:10:42.957057 ip-10-0-136-165 kubenswrapper[2526]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 18:10:42.957057 ip-10-0-136-165 kubenswrapper[2526]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 18:10:42.957057 ip-10-0-136-165 kubenswrapper[2526]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 18:10:42.957057 ip-10-0-136-165 kubenswrapper[2526]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 18:10:42.957666 ip-10-0-136-165 kubenswrapper[2526]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 18:10:42.958509 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.958407 2526 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 18:10:42.961523 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961508 2526 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 18:10:42.961523 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961523 2526 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961526 2526 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961531 2526 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961534 2526 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961537 2526 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961540 2526 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961543 2526 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961547 2526 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961555 2526 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961557 2526 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961560 2526 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961563 2526 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961565 2526 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961568 2526 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961571 2526 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961573 2526 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961576 2526 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961579 2526 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961582 2526 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961584 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 18:10:42.961586 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961587 2526 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961589 2526 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961592 2526 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961595 2526 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961598 2526 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961601 2526 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961603 2526 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961606 2526 feature_gate.go:328] unrecognized feature gate: Example Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961608 2526 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961611 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961614 2526 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961616 2526 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961621 2526 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961624 2526 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961627 2526 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961630 2526 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961632 2526 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961635 2526 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961638 2526 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961640 2526 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 18:10:42.962076 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961643 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961646 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961648 2526 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961651 2526 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961654 2526 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961656 2526 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961659 2526 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961661 2526 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961664 2526 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961666 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961668 2526 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961671 2526 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961673 2526 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961677 2526 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961681 2526 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961684 2526 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961687 2526 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961690 2526 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961693 2526 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 18:10:42.962590 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961696 2526 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961698 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961701 2526 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961704 2526 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961706 2526 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961709 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961712 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961714 2526 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961716 2526 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961719 2526 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961722 2526 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961725 2526 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961728 2526 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961731 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961733 2526 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961736 2526 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961738 2526 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961741 2526 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961744 2526 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961746 2526 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 18:10:42.963063 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961748 2526 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961751 2526 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961753 2526 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961756 2526 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961758 2526 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961761 2526 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961763 2526 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961767 2526 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961769 2526 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961772 2526 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961775 2526 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961777 2526 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961780 2526 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961782 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961786 2526 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961790 2526 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961793 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961795 2526 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.961797 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 18:10:42.963580 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962466 2526 feature_gate.go:328] unrecognized feature gate: Example Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962472 2526 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962476 2526 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962478 2526 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962481 2526 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962484 2526 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962486 2526 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962489 2526 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962492 2526 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962494 2526 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962497 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962501 2526 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962503 2526 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962506 2526 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962508 2526 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962511 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962514 2526 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962517 2526 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962519 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962523 2526 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 18:10:42.964049 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962525 2526 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962528 2526 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962530 2526 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962533 2526 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962535 2526 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962538 2526 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962541 2526 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962543 2526 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962546 2526 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962548 2526 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962551 2526 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962553 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962556 2526 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962559 2526 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962561 2526 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962564 2526 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962566 2526 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962569 2526 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962572 2526 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962575 2526 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 18:10:42.964550 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962578 2526 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962580 2526 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962583 2526 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962586 2526 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962588 2526 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962591 2526 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962594 2526 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962596 2526 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962599 2526 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962602 2526 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962605 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962608 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962611 2526 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962614 2526 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962616 2526 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962619 2526 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962621 2526 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962624 2526 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962627 2526 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 18:10:42.965040 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962629 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962633 2526 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962637 2526 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962640 2526 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962643 2526 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962646 2526 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962648 2526 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962651 2526 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962653 2526 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962656 2526 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962660 2526 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962664 2526 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962668 2526 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962672 2526 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962674 2526 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962677 2526 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962680 2526 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962683 2526 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962686 2526 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 18:10:42.965513 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962688 2526 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962691 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962694 2526 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962696 2526 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962699 2526 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962702 2526 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962704 2526 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962707 2526 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962711 2526 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962714 2526 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962717 2526 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962720 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962723 2526 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962726 2526 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962728 2526 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962731 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962733 2526 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962736 2526 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962738 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962741 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 18:10:42.965988 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.962743 2526 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963105 2526 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963110 2526 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963113 2526 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963116 2526 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963122 2526 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963125 2526 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963127 2526 feature_gate.go:328] unrecognized feature gate: Example Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963130 2526 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963133 2526 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963135 2526 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963138 2526 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963140 2526 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963143 2526 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963146 2526 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963148 2526 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963151 2526 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963154 2526 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963156 2526 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963159 2526 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 18:10:42.966503 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963161 2526 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963164 2526 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963167 2526 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963169 2526 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963172 2526 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963175 2526 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963177 2526 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963180 2526 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963182 2526 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963185 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963188 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963190 2526 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963193 2526 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963195 2526 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963198 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963200 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963203 2526 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963205 2526 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963208 2526 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963211 2526 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 18:10:42.966997 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963214 2526 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963217 2526 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963221 2526 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963225 2526 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963228 2526 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963231 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963235 2526 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963239 2526 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963242 2526 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963245 2526 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963248 2526 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963252 2526 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963256 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963259 2526 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963261 2526 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963264 2526 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963267 2526 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963269 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963272 2526 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 18:10:42.967481 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963274 2526 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963277 2526 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963280 2526 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963282 2526 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963285 2526 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963287 2526 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963292 2526 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963295 2526 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963298 2526 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963301 2526 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963304 2526 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963307 2526 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963311 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963314 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963317 2526 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963320 2526 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963323 2526 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963325 2526 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963328 2526 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 18:10:42.967942 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963331 2526 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963334 2526 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963337 2526 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963340 2526 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963342 2526 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963345 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963347 2526 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963350 2526 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963353 2526 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963355 2526 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963358 2526 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963360 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963364 2526 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963367 2526 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963369 2526 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963372 2526 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963374 2526 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963377 2526 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963379 2526 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963382 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 18:10:42.968432 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963384 2526 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.963387 2526 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963459 2526 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963465 2526 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963472 2526 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963476 2526 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963481 2526 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963484 2526 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963488 2526 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963492 2526 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963496 2526 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963498 2526 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963502 2526 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963506 2526 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963510 2526 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963513 2526 flags.go:64] FLAG: --cgroup-root="" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963516 2526 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963519 2526 flags.go:64] FLAG: --client-ca-file="" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963522 2526 flags.go:64] FLAG: --cloud-provider="external" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963525 2526 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963529 2526 flags.go:64] FLAG: --cluster-domain="" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963531 2526 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963534 2526 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963537 2526 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 18:10:42.968902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963542 2526 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963545 2526 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963548 2526 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963551 2526 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963555 2526 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963558 2526 flags.go:64] FLAG: --contention-profiling="false" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963561 2526 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963564 2526 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963567 2526 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963569 2526 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963581 2526 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963585 2526 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963588 2526 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963591 2526 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963594 2526 flags.go:64] FLAG: --enable-server="true" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963598 2526 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963602 2526 flags.go:64] FLAG: --event-burst="100" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963606 2526 flags.go:64] FLAG: --event-qps="50" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963609 2526 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963612 2526 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963614 2526 flags.go:64] FLAG: --eviction-hard="" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963619 2526 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963622 2526 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963624 2526 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963627 2526 flags.go:64] FLAG: --eviction-soft="" Aug 04 18:10:42.969497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963630 2526 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963633 2526 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963636 2526 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963639 2526 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963642 2526 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963644 2526 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963647 2526 flags.go:64] FLAG: --feature-gates="" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963652 2526 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963655 2526 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963658 2526 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963661 2526 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963664 2526 flags.go:64] FLAG: --healthz-port="10248" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963667 2526 flags.go:64] FLAG: --help="false" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963670 2526 flags.go:64] FLAG: --hostname-override="ip-10-0-136-165.ec2.internal" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963673 2526 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963676 2526 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963679 2526 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963682 2526 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963685 2526 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963688 2526 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963691 2526 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963694 2526 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963696 2526 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 18:10:42.970109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963700 2526 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963703 2526 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963706 2526 flags.go:64] FLAG: --kube-reserved="" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963709 2526 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963712 2526 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963715 2526 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963718 2526 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963720 2526 flags.go:64] FLAG: --lock-file="" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963723 2526 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963726 2526 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963729 2526 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963734 2526 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963737 2526 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963740 2526 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963742 2526 flags.go:64] FLAG: --logging-format="text" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963745 2526 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963753 2526 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963757 2526 flags.go:64] FLAG: --manifest-url="" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963759 2526 flags.go:64] FLAG: --manifest-url-header="" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963764 2526 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963767 2526 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963771 2526 flags.go:64] FLAG: --max-pods="110" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963774 2526 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963777 2526 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963780 2526 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 18:10:42.970714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963783 2526 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963786 2526 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963788 2526 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963791 2526 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963798 2526 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963801 2526 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963804 2526 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963807 2526 flags.go:64] FLAG: --pod-cidr="" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963811 2526 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963816 2526 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963819 2526 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963822 2526 flags.go:64] FLAG: --pods-per-core="0" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963825 2526 flags.go:64] FLAG: --port="10250" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963828 2526 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963831 2526 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a6bc13da026dfffc" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963834 2526 flags.go:64] FLAG: --qos-reserved="" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963837 2526 flags.go:64] FLAG: --read-only-port="10255" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963840 2526 flags.go:64] FLAG: --register-node="true" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963843 2526 flags.go:64] FLAG: --register-with-taints="" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963846 2526 flags.go:64] FLAG: --registry-burst="10" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963849 2526 flags.go:64] FLAG: --registry-qps="5" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963852 2526 flags.go:64] FLAG: --reserved-cpus="" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963855 2526 flags.go:64] FLAG: --reserved-memory="" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963860 2526 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963863 2526 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 18:10:42.971325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963866 2526 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963869 2526 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963872 2526 flags.go:64] FLAG: --runonce="false" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963874 2526 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963877 2526 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963880 2526 flags.go:64] FLAG: --seccomp-default="false" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963883 2526 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963886 2526 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963889 2526 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963892 2526 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963895 2526 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963898 2526 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963901 2526 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963903 2526 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963906 2526 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963909 2526 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963913 2526 flags.go:64] FLAG: --system-cgroups="" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963915 2526 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963921 2526 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963923 2526 flags.go:64] FLAG: --tls-cert-file="" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963926 2526 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963930 2526 flags.go:64] FLAG: --tls-min-version="" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963933 2526 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963936 2526 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963939 2526 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 18:10:42.971941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963941 2526 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963944 2526 flags.go:64] FLAG: --v="2" Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963948 2526 flags.go:64] FLAG: --version="false" Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963951 2526 flags.go:64] FLAG: --vmodule="" Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963960 2526 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.963979 2526 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964089 2526 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964093 2526 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964097 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964100 2526 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964103 2526 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964106 2526 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964108 2526 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964111 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964114 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964117 2526 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964119 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964122 2526 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964124 2526 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964127 2526 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964129 2526 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 18:10:42.972557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964132 2526 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964134 2526 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964137 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964140 2526 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964142 2526 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964145 2526 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964150 2526 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964153 2526 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964156 2526 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964158 2526 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964161 2526 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964163 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964166 2526 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964169 2526 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964171 2526 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964174 2526 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964178 2526 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964180 2526 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964183 2526 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964185 2526 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 18:10:42.973126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964189 2526 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964193 2526 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964196 2526 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964199 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964202 2526 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964205 2526 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964207 2526 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964210 2526 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964213 2526 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964215 2526 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964218 2526 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964220 2526 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964223 2526 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964226 2526 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964228 2526 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964231 2526 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964234 2526 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964237 2526 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964240 2526 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964243 2526 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 18:10:42.973626 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964246 2526 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964249 2526 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964253 2526 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964256 2526 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964259 2526 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964262 2526 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964264 2526 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964267 2526 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964271 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964274 2526 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964276 2526 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964280 2526 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964282 2526 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964285 2526 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964287 2526 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964290 2526 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964293 2526 feature_gate.go:328] unrecognized feature gate: Example Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964295 2526 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964298 2526 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 18:10:42.974382 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964300 2526 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964303 2526 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964305 2526 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964308 2526 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964310 2526 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964313 2526 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964315 2526 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964318 2526 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964320 2526 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964323 2526 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964326 2526 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964330 2526 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964332 2526 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964335 2526 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964338 2526 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964340 2526 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964343 2526 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964346 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964348 2526 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964351 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 18:10:42.974923 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964353 2526 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 18:10:42.975440 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964357 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 18:10:42.975440 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964360 2526 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 18:10:42.975440 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964362 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 18:10:42.975440 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.964365 2526 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 18:10:42.975440 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.964922 2526 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 18:10:42.975440 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.971507 2526 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 18:10:42.975440 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.971522 2526 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 18:10:42.975440 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.971537 2526 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 18:10:42.975440 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.971543 2526 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 18:10:42.975440 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971602 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 18:10:42.975440 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971608 2526 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 18:10:42.975440 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971611 2526 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 18:10:42.975440 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971614 2526 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 18:10:42.975440 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971616 2526 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 18:10:42.975440 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971619 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971622 2526 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971625 2526 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971628 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971630 2526 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971633 2526 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971636 2526 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971639 2526 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971641 2526 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971644 2526 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971646 2526 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971649 2526 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971651 2526 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971654 2526 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971656 2526 feature_gate.go:328] unrecognized feature gate: Example Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971659 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971662 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971664 2526 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971667 2526 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 18:10:42.975977 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971669 2526 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971672 2526 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971675 2526 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971678 2526 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971680 2526 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971684 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971687 2526 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971690 2526 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971693 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971695 2526 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971699 2526 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971703 2526 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971706 2526 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971709 2526 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971711 2526 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971714 2526 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971716 2526 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971719 2526 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971721 2526 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 18:10:42.976444 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971724 2526 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971727 2526 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971729 2526 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971731 2526 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971734 2526 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971736 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971739 2526 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971742 2526 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971745 2526 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971747 2526 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971750 2526 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971752 2526 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971755 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971757 2526 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971759 2526 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971762 2526 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971765 2526 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971767 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971771 2526 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971774 2526 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 18:10:42.976904 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971777 2526 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971779 2526 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971782 2526 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971785 2526 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971787 2526 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971790 2526 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971793 2526 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971795 2526 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971797 2526 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971800 2526 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971803 2526 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971805 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971808 2526 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971810 2526 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971813 2526 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971815 2526 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971818 2526 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971820 2526 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971823 2526 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971825 2526 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 18:10:42.977404 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971828 2526 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 18:10:42.977894 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971831 2526 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 18:10:42.977894 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971834 2526 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 18:10:42.977894 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971838 2526 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 18:10:42.977894 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971841 2526 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 18:10:42.977894 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971845 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 18:10:42.977894 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971848 2526 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 18:10:42.977894 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971850 2526 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 18:10:42.977894 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971853 2526 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 18:10:42.977894 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971856 2526 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 18:10:42.977894 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971860 2526 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 18:10:42.977894 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971863 2526 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 18:10:42.977894 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971866 2526 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 18:10:42.977894 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971869 2526 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 18:10:42.977894 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971872 2526 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 18:10:42.977894 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.971875 2526 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 18:10:42.978363 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.971880 2526 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 18:10:42.978363 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972011 2526 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 18:10:42.978363 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972017 2526 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 18:10:42.978363 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972020 2526 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 18:10:42.978363 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972023 2526 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 18:10:42.978363 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972026 2526 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 18:10:42.978363 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972028 2526 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 18:10:42.978363 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972031 2526 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 18:10:42.978363 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972033 2526 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 18:10:42.978363 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972037 2526 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 18:10:42.978363 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972041 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 18:10:42.978363 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972044 2526 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 18:10:42.978363 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972046 2526 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 18:10:42.978363 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972049 2526 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972052 2526 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972055 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972058 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972061 2526 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972063 2526 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972066 2526 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972068 2526 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972071 2526 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972073 2526 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972076 2526 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972078 2526 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972081 2526 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972084 2526 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972087 2526 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972089 2526 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972092 2526 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972095 2526 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972098 2526 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972100 2526 feature_gate.go:328] unrecognized feature gate: Example Aug 04 18:10:42.978708 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972102 2526 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972106 2526 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972110 2526 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972113 2526 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972116 2526 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972118 2526 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972121 2526 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972123 2526 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972126 2526 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972128 2526 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972131 2526 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972133 2526 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972135 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972138 2526 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972141 2526 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972143 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972146 2526 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972148 2526 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972151 2526 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972153 2526 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 18:10:42.979240 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972156 2526 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972158 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972161 2526 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972163 2526 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972166 2526 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972169 2526 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972171 2526 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972174 2526 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972176 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972179 2526 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972182 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972184 2526 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972187 2526 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972189 2526 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972192 2526 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972194 2526 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972197 2526 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972199 2526 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972202 2526 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972204 2526 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 18:10:42.979730 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972207 2526 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972209 2526 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972212 2526 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972214 2526 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972217 2526 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972219 2526 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972222 2526 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972224 2526 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972226 2526 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972229 2526 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972231 2526 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972235 2526 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972239 2526 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972242 2526 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972245 2526 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972248 2526 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972250 2526 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972253 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972256 2526 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 18:10:42.980242 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972259 2526 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 18:10:42.980694 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972261 2526 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 18:10:42.980694 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972264 2526 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 18:10:42.980694 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972267 2526 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 18:10:42.980694 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972270 2526 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 18:10:42.980694 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972272 2526 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 18:10:42.980694 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972275 2526 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 18:10:42.980694 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:42.972278 2526 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 18:10:42.980694 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.972283 2526 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 18:10:42.980694 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.972757 2526 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 18:10:42.980694 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.975610 2526 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 18:10:42.980694 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.976894 2526 server.go:1019] "Starting client certificate rotation" Aug 04 18:10:42.980694 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.976994 2526 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 18:10:42.980694 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.977025 2526 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 18:10:42.981034 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.979019 2526 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 18:10:42.981034 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.980099 2526 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 18:10:42.987783 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.987766 2526 log.go:25] "Validated CRI v1 runtime API" Aug 04 18:10:42.989720 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.989707 2526 log.go:25] "Validated CRI v1 image API" Aug 04 18:10:42.993314 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.993299 2526 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 18:10:42.995427 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.995408 2526 fs.go:135] Filesystem UUIDs: map[28ff6a91-6f2b-422d-be0f-0538ccf27606:/dev/nvme0n1p4 69948dbb-56af-42d8-8d14-db2bdf57e036:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Aug 04 18:10:42.995497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:42.995426 2526 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 18:10:43.000596 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.000474 2526 manager.go:216] Machine: {Timestamp:2026-08-04 18:10:42.999995069 +0000 UTC m=+0.251277606 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099672 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25307191425380a4adb04d46ea6095 SystemUUID:ec253071-9142-5380-a4ad-b04d46ea6095 BootID:d8f79cd5-c973-4b4b-8e89-c3986aef9362 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cd:95:8f:9a:01 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cd:95:8f:9a:01 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:d5:25:d6:a3:cd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 18:10:43.000596 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.000593 2526 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 18:10:43.000696 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.000668 2526 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 18:10:43.003083 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.003056 2526 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 18:10:43.003212 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.003089 2526 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-165.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 18:10:43.003668 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.003656 2526 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 18:10:43.003668 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.003670 2526 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 18:10:43.003744 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.003677 2526 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 18:10:43.004479 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.004466 2526 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 18:10:43.005347 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.005337 2526 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 18:10:43.005706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.005689 2526 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 18:10:43.009778 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.009766 2526 state_mem.go:36] "Initialized new in-memory state store" Aug 04 18:10:43.009911 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.009895 2526 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 18:10:43.010011 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.009926 2526 kubelet.go:486] "Attempting to sync node with API server" Aug 04 18:10:43.010011 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.009936 2526 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 18:10:43.010011 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.009949 2526 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 18:10:43.010011 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.009958 2526 kubelet.go:398] "Adding apiserver pod source" Aug 04 18:10:43.010011 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.009979 2526 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 18:10:43.010338 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.010324 2526 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 18:10:43.010395 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.010347 2526 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 18:10:43.012424 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.012400 2526 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 18:10:43.014115 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.014098 2526 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 18:10:43.014186 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.014130 2526 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 18:10:43.015478 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.015464 2526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 18:10:43.015537 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.015488 2526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 18:10:43.015537 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.015497 2526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 18:10:43.015537 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.015506 2526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 18:10:43.015537 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.015518 2526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 18:10:43.015537 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.015536 2526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 18:10:43.015827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.015550 2526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 18:10:43.015827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.015558 2526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 18:10:43.015827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.015582 2526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 18:10:43.015827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.015592 2526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 18:10:43.015827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.015605 2526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 18:10:43.015827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.015614 2526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 18:10:43.016490 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.016473 2526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 18:10:43.016568 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.016507 2526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 18:10:43.019896 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.019882 2526 server.go:1268] "Started kubelet" Aug 04 18:10:43.020020 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.019959 2526 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 18:10:43.020087 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.020043 2526 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 18:10:43.020289 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.020158 2526 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 18:10:43.020756 ip-10-0-136-165 systemd[1]: Started Kubernetes Kubelet. Aug 04 18:10:43.021407 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.021288 2526 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 18:10:43.021838 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.021814 2526 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-165.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 18:10:43.021919 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.021889 2526 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 18:10:43.021989 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.021918 2526 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-165.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 18:10:43.022642 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.022626 2526 server.go:310] "Adding debug handlers to kubelet server" Aug 04 18:10:43.024650 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.024624 2526 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 18:10:43.025442 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.025424 2526 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 18:10:43.025442 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.025434 2526 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 18:10:43.025562 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.025508 2526 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:43.026051 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.026022 2526 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 18:10:43.026136 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.026058 2526 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 18:10:43.026136 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.026068 2526 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 18:10:43.026222 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.026204 2526 reconstruct.go:97] "Volume reconstruction finished" Aug 04 18:10:43.026222 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.026213 2526 reconciler.go:29] "Reconciler: start to sync state" Aug 04 18:10:43.026462 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.026443 2526 factory.go:55] Registering systemd factory Aug 04 18:10:43.026531 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.026473 2526 factory.go:223] Registration of the systemd container factory successfully Aug 04 18:10:43.026701 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.026685 2526 factory.go:153] Registering CRI-O factory Aug 04 18:10:43.026701 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.026700 2526 factory.go:223] Registration of the crio container factory successfully Aug 04 18:10:43.026838 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.026757 2526 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 18:10:43.026838 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.026782 2526 factory.go:103] Registering Raw factory Aug 04 18:10:43.026838 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.026797 2526 manager.go:1196] Started watching for new ooms in manager Aug 04 18:10:43.027186 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.027174 2526 manager.go:349] Starting recovery of all containers Aug 04 18:10:43.028557 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.028534 2526 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-165.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 18:10:43.028872 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.028852 2526 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 18:10:43.029442 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.028509 2526 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-165.ec2.internal.18c8ac9d1b467c1e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-165.ec2.internal,UID:ip-10-0-136-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-165.ec2.internal,},FirstTimestamp:2026-08-04 18:10:43.019848734 +0000 UTC m=+0.271131275,LastTimestamp:2026-08-04 18:10:43.019848734 +0000 UTC m=+0.271131275,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-165.ec2.internal,}" Aug 04 18:10:43.039337 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.039320 2526 manager.go:354] Recovery completed Aug 04 18:10:43.044958 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.044945 2526 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 18:10:43.046739 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.046721 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientMemory" Aug 04 18:10:43.046836 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.046751 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 18:10:43.046836 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.046762 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientPID" Aug 04 18:10:43.047297 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.047277 2526 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 18:10:43.047297 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.047290 2526 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 18:10:43.047297 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.047294 2526 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 18:10:43.047472 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.047309 2526 state_mem.go:36] "Initialized new in-memory state store" Aug 04 18:10:43.047927 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.047913 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientMemory" Aug 04 18:10:43.048027 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.047942 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 18:10:43.048027 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.047956 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientPID" Aug 04 18:10:43.049082 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.049001 2526 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-165.ec2.internal.18c8ac9d1ce0cb1a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-165.ec2.internal,UID:ip-10-0-136-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-165.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-165.ec2.internal,},FirstTimestamp:2026-08-04 18:10:43.046738714 +0000 UTC m=+0.298021254,LastTimestamp:2026-08-04 18:10:43.046738714 +0000 UTC m=+0.298021254,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-165.ec2.internal,}" Aug 04 18:10:43.049502 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.049489 2526 policy_none.go:49] "None policy: Start" Aug 04 18:10:43.049502 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.049504 2526 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 18:10:43.049587 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.049515 2526 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 18:10:43.051522 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.051507 2526 policy_none.go:47] "Start" Aug 04 18:10:43.057178 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.057102 2526 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-165.ec2.internal.18c8ac9d1ce1103c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-165.ec2.internal,UID:ip-10-0-136-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-165.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-165.ec2.internal,},FirstTimestamp:2026-08-04 18:10:43.046756412 +0000 UTC m=+0.298038952,LastTimestamp:2026-08-04 18:10:43.046756412 +0000 UTC m=+0.298038952,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-165.ec2.internal,}" Aug 04 18:10:43.059331 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.059274 2526 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-165.ec2.internal.18c8ac9d1ce1346b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-165.ec2.internal,UID:ip-10-0-136-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-165.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-165.ec2.internal,},FirstTimestamp:2026-08-04 18:10:43.046765675 +0000 UTC m=+0.298048216,LastTimestamp:2026-08-04 18:10:43.046765675 +0000 UTC m=+0.298048216,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-165.ec2.internal,}" Aug 04 18:10:43.066298 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.066240 2526 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-165.ec2.internal.18c8ac9d1ce0cb1a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-165.ec2.internal.18c8ac9d1ce0cb1a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-165.ec2.internal,UID:ip-10-0-136-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-165.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-165.ec2.internal,},FirstTimestamp:2026-08-04 18:10:43.046738714 +0000 UTC m=+0.298021254,LastTimestamp:2026-08-04 18:10:43.047926697 +0000 UTC m=+0.299209239,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-165.ec2.internal,}" Aug 04 18:10:43.071662 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.071643 2526 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6z7vz" Aug 04 18:10:43.072287 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.072216 2526 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-165.ec2.internal.18c8ac9d1ce1103c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-165.ec2.internal.18c8ac9d1ce1103c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-165.ec2.internal,UID:ip-10-0-136-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-165.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-165.ec2.internal,},FirstTimestamp:2026-08-04 18:10:43.046756412 +0000 UTC m=+0.298038952,LastTimestamp:2026-08-04 18:10:43.047948865 +0000 UTC m=+0.299231404,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-165.ec2.internal,}" Aug 04 18:10:43.075628 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.075563 2526 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-165.ec2.internal.18c8ac9d1ce1346b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-165.ec2.internal.18c8ac9d1ce1346b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-165.ec2.internal,UID:ip-10-0-136-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-165.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-165.ec2.internal,},FirstTimestamp:2026-08-04 18:10:43.046765675 +0000 UTC m=+0.298048216,LastTimestamp:2026-08-04 18:10:43.047976224 +0000 UTC m=+0.299258766,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-165.ec2.internal,}" Aug 04 18:10:43.083332 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.083317 2526 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6z7vz" Aug 04 18:10:43.086541 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.086526 2526 manager.go:337] "Starting Device Plugin manager" Aug 04 18:10:43.086608 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.086561 2526 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 18:10:43.086608 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.086575 2526 server.go:87] "Starting device plugin registration server" Aug 04 18:10:43.086837 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.086821 2526 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 18:10:43.086901 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.086846 2526 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 18:10:43.086945 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.086936 2526 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 18:10:43.087033 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.087019 2526 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 18:10:43.087090 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.087033 2526 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 18:10:43.087532 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.087512 2526 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 18:10:43.087599 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.087569 2526 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:43.090953 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.090895 2526 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-165.ec2.internal.18c8ac9d1f6099ed default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-165.ec2.internal,UID:ip-10-0-136-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-136-165.ec2.internal,},FirstTimestamp:2026-08-04 18:10:43.088669165 +0000 UTC m=+0.339951694,LastTimestamp:2026-08-04 18:10:43.088669165 +0000 UTC m=+0.339951694,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-165.ec2.internal,}" Aug 04 18:10:43.173188 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.173145 2526 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 18:10:43.174398 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.174378 2526 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 18:10:43.174462 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.174420 2526 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 18:10:43.174462 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.174447 2526 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 18:10:43.174563 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.174482 2526 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 18:10:43.187312 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.187289 2526 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 18:10:43.187728 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.187709 2526 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 18:10:43.188052 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.188038 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientMemory" Aug 04 18:10:43.188131 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.188066 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 18:10:43.188131 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.188076 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientPID" Aug 04 18:10:43.188131 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.188104 2526 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.198055 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.198039 2526 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.198104 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.198059 2526 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-165.ec2.internal\": node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:43.216718 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.216662 2526 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:43.275529 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.275500 2526 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-165.ec2.internal"] Aug 04 18:10:43.275641 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.275568 2526 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 18:10:43.276394 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.276378 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientMemory" Aug 04 18:10:43.276484 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.276413 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 18:10:43.276484 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.276428 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientPID" Aug 04 18:10:43.278850 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.278833 2526 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 18:10:43.278973 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.278957 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.279036 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.279016 2526 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 18:10:43.279558 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.279540 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientMemory" Aug 04 18:10:43.279628 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.279567 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 18:10:43.279628 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.279541 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientMemory" Aug 04 18:10:43.279628 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.279603 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 18:10:43.279628 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.279624 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientPID" Aug 04 18:10:43.279752 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.279578 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientPID" Aug 04 18:10:43.282314 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.282295 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.282403 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.282329 2526 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 18:10:43.283101 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.283086 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientMemory" Aug 04 18:10:43.283186 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.283115 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 18:10:43.283186 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.283126 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientPID" Aug 04 18:10:43.308852 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.308834 2526 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-165.ec2.internal\" not found" node="ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.313433 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.313417 2526 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-165.ec2.internal\" not found" node="ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.317441 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.317423 2526 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:43.327951 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.327929 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c07c5fa66b82b163d24a10ef1ee49cd1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal\" (UID: \"c07c5fa66b82b163d24a10ef1ee49cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.328037 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.327962 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c07c5fa66b82b163d24a10ef1ee49cd1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal\" (UID: \"c07c5fa66b82b163d24a10ef1ee49cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.328037 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.327994 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a598d320185d24ca207d357f68b8dce3-config\") pod \"kube-apiserver-proxy-ip-10-0-136-165.ec2.internal\" (UID: \"a598d320185d24ca207d357f68b8dce3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.417747 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.417729 2526 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:43.428595 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.428578 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c07c5fa66b82b163d24a10ef1ee49cd1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal\" (UID: \"c07c5fa66b82b163d24a10ef1ee49cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.428661 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.428610 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c07c5fa66b82b163d24a10ef1ee49cd1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal\" (UID: \"c07c5fa66b82b163d24a10ef1ee49cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.428661 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.428633 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a598d320185d24ca207d357f68b8dce3-config\") pod \"kube-apiserver-proxy-ip-10-0-136-165.ec2.internal\" (UID: \"a598d320185d24ca207d357f68b8dce3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.428768 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.428665 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c07c5fa66b82b163d24a10ef1ee49cd1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal\" (UID: \"c07c5fa66b82b163d24a10ef1ee49cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.428768 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.428687 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c07c5fa66b82b163d24a10ef1ee49cd1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal\" (UID: \"c07c5fa66b82b163d24a10ef1ee49cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.428768 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.428753 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a598d320185d24ca207d357f68b8dce3-config\") pod \"kube-apiserver-proxy-ip-10-0-136-165.ec2.internal\" (UID: \"a598d320185d24ca207d357f68b8dce3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.518375 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.518323 2526 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:43.617369 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.617339 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.617451 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.617347 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-165.ec2.internal" Aug 04 18:10:43.618518 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.618504 2526 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:43.719509 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.719479 2526 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:43.826096 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.826040 2526 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:43.932554 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:43.932525 2526 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:43.972572 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.972551 2526 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 18:10:43.977018 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.977004 2526 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 18:10:43.977135 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.977118 2526 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 18:10:43.977190 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:43.977121 2526 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 18:10:44.032796 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:44.032766 2526 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:44.069151 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:44.069128 2526 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 18:10:44.070192 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:44.070176 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientMemory" Aug 04 18:10:44.070278 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:44.070210 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 18:10:44.070278 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:44.070226 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeHasSufficientPID" Aug 04 18:10:44.088006 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:44.087926 2526 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 18:05:43 +0000 UTC" deadline="2028-05-01 02:47:42.796694602 +0000 UTC" Aug 04 18:10:44.088006 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:44.087950 2526 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15248h36m58.70874738s" Aug 04 18:10:44.133333 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:44.133308 2526 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:44.234518 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:44.234492 2526 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:44.280367 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:44.280327 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda598d320185d24ca207d357f68b8dce3.slice/crio-370006406a23597eb438cd61e3bfe04363a447ac114b27c52ab57dd616fb27b8 WatchSource:0}: Error finding container 370006406a23597eb438cd61e3bfe04363a447ac114b27c52ab57dd616fb27b8: Status 404 returned error can't find the container with id 370006406a23597eb438cd61e3bfe04363a447ac114b27c52ab57dd616fb27b8 Aug 04 18:10:44.288181 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:44.288162 2526 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:10:44.293501 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:44.293479 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07c5fa66b82b163d24a10ef1ee49cd1.slice/crio-8199018e6e2e789770e981cf0211a58e6cfe0d47802a81266fe6d77c07e4ccee WatchSource:0}: Error finding container 8199018e6e2e789770e981cf0211a58e6cfe0d47802a81266fe6d77c07e4ccee: Status 404 returned error can't find the container with id 8199018e6e2e789770e981cf0211a58e6cfe0d47802a81266fe6d77c07e4ccee Aug 04 18:10:44.334852 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:44.334821 2526 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:44.384090 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:44.384023 2526 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 18:10:44.435126 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:44.435097 2526 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-165.ec2.internal\" not found" Aug 04 18:10:44.477419 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:44.477399 2526 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 18:10:44.526293 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:44.526266 2526 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal" Aug 04 18:10:44.540004 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:44.539982 2526 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 18:10:44.540704 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:44.540690 2526 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-165.ec2.internal" Aug 04 18:10:44.556874 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:44.556860 2526 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 18:10:44.845670 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:44.845603 2526 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 18:10:45.010854 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.010825 2526 apiserver.go:52] "Watching apiserver" Aug 04 18:10:45.018071 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.018037 2526 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 18:10:45.018582 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.018558 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-r59lx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm","openshift-image-registry/node-ca-p9xc7","openshift-multus/multus-6m9tw","openshift-network-diagnostics/network-check-target-qmq4s","openshift-network-operator/iptables-alerter-krqrq","openshift-ovn-kubernetes/ovnkube-node-7ns2w","kube-system/konnectivity-agent-845wp","kube-system/kube-apiserver-proxy-ip-10-0-136-165.ec2.internal","openshift-cluster-node-tuning-operator/tuned-lp29p","openshift-dns/node-resolver-rvt4h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal","openshift-multus/multus-additional-cni-plugins-g2zhc","openshift-multus/network-metrics-daemon-ntsjj"] Aug 04 18:10:45.021721 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.021315 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-845wp" Aug 04 18:10:45.023878 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.023854 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.024564 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.024545 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zf8p8\"" Aug 04 18:10:45.025315 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.025296 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 18:10:45.025586 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.025566 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 18:10:45.025787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.025768 2526 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 18:10:45.026639 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.026621 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 18:10:45.029399 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.029377 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 18:10:45.029575 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.029536 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v2qq4\"" Aug 04 18:10:45.029647 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.029625 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 18:10:45.029801 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.029783 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:45.029896 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.029857 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:10:45.030455 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.030424 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.033102 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.033083 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:10:45.033204 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.033156 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:10:45.033924 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.033901 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 18:10:45.034325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.034233 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 18:10:45.034325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.034240 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-52wr9\"" Aug 04 18:10:45.034713 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.034423 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 18:10:45.034713 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.034666 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 18:10:45.035204 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.035184 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-krqrq" Aug 04 18:10:45.040069 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.039696 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-var-lib-cni-multus\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.040069 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.039737 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-system-cni-dir\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.040069 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.039769 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ceb0e538-c878-429c-b09b-bad693514c20-cni-binary-copy\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.040069 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.039801 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df338c86-7a60-48e5-ba0a-79a483d7275c-konnectivity-ca\") pod \"konnectivity-agent-845wp\" (UID: \"df338c86-7a60-48e5-ba0a-79a483d7275c\") " pod="kube-system/konnectivity-agent-845wp" Aug 04 18:10:45.040069 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.039835 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgrfh\" (UniqueName: \"kubernetes.io/projected/13c6d111-9af3-4928-a9fc-5533cae22fa4-kube-api-access-wgrfh\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.040069 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.039864 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-multus-cni-dir\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.040069 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.039890 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-run-k8s-cni-cncf-io\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.040069 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.039920 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-etc-kubernetes\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.040069 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.039954 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.040069 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.040002 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-socket-dir\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.040069 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.040031 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-etc-selinux\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.040775 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.040697 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-sys-fs\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.040740 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-var-lib-kubelet\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.040888 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-hostroot\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.040920 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-run-multus-certs\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.040949 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ceb0e538-c878-429c-b09b-bad693514c20-multus-daemon-config\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.040973 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df338c86-7a60-48e5-ba0a-79a483d7275c-agent-certs\") pod \"konnectivity-agent-845wp\" (UID: \"df338c86-7a60-48e5-ba0a-79a483d7275c\") " pod="kube-system/konnectivity-agent-845wp" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.041003 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-cnibin\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.041032 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-os-release\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.041079 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-multus-socket-dir-parent\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.041130 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-run-netns\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.041175 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-registration-dir\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.041209 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-device-dir\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.041242 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs\") pod \"network-metrics-daemon-ntsjj\" (UID: \"11d07db5-806d-44e7-af32-be9ae461cbae\") " pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.041285 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vnsc\" (UniqueName: \"kubernetes.io/projected/11d07db5-806d-44e7-af32-be9ae461cbae-kube-api-access-4vnsc\") pod \"network-metrics-daemon-ntsjj\" (UID: \"11d07db5-806d-44e7-af32-be9ae461cbae\") " pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.041337 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgjng\" (UniqueName: \"kubernetes.io/projected/ceb0e538-c878-429c-b09b-bad693514c20-kube-api-access-pgjng\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.041390 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-var-lib-cni-bin\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.041433 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-multus-conf-dir\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.042134 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p6v29\"" Aug 04 18:10:45.042932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.040503 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.043961 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.042596 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:45.043961 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.042669 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 18:10:45.043961 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.042695 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:10:45.043961 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.042930 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 18:10:45.043961 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.043802 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 18:10:45.047871 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.047036 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.048242 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.048218 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 18:10:45.048463 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.048447 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 18:10:45.049270 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.048843 2526 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 18:10:45.049270 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.048909 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 18:10:45.049270 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.049177 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rvt4h" Aug 04 18:10:45.050063 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.050028 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 18:10:45.050154 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.050109 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 18:10:45.050286 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.050267 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-988kc\"" Aug 04 18:10:45.050405 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.050337 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 18:10:45.051309 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.051289 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 18:10:45.051818 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.051797 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 18:10:45.051919 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.051901 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.052023 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.052006 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 18:10:45.054817 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.054798 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p9xc7" Aug 04 18:10:45.056181 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.056161 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-twwwp\"" Aug 04 18:10:45.056719 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.056696 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 18:10:45.056972 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.056954 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 18:10:45.057225 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.057206 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d2ztn\"" Aug 04 18:10:45.057429 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.057411 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j2n4w\"" Aug 04 18:10:45.057638 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.057611 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 18:10:45.060799 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.060573 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 18:10:45.060799 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.060601 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 18:10:45.060994 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.060825 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w2w9k\"" Aug 04 18:10:45.060994 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.060859 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 18:10:45.087294 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.087232 2526 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bvkds" Aug 04 18:10:45.099999 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.099898 2526 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bvkds" Aug 04 18:10:45.127318 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.127282 2526 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 18:10:45.142163 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142140 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-var-lib-cni-bin\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.142253 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142176 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-multus-conf-dir\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.142253 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142219 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9nv6\" (UniqueName: \"kubernetes.io/projected/884f5581-c681-4785-a065-fa174e052347-kube-api-access-f9nv6\") pod \"node-ca-p9xc7\" (UID: \"884f5581-c681-4785-a065-fa174e052347\") " pod="openshift-image-registry/node-ca-p9xc7" Aug 04 18:10:45.142253 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142246 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-system-cni-dir\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.142380 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142271 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ceb0e538-c878-429c-b09b-bad693514c20-cni-binary-copy\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.142380 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142297 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-sysconfig\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.142380 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142322 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-systemd-units\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.142380 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142352 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df338c86-7a60-48e5-ba0a-79a483d7275c-konnectivity-ca\") pod \"konnectivity-agent-845wp\" (UID: \"df338c86-7a60-48e5-ba0a-79a483d7275c\") " pod="kube-system/konnectivity-agent-845wp" Aug 04 18:10:45.142380 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142377 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-multus-cni-dir\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.142592 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142411 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-run-k8s-cni-cncf-io\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.142592 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142436 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-run-ovn\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.142592 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142486 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-multus-conf-dir\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.142592 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142501 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.142592 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142522 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-socket-dir\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.142592 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142546 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-hostroot\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.142592 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142554 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-system-cni-dir\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.142592 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142578 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-kubelet\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.142990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142605 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-ovnkube-config\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.142990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142682 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-var-lib-cni-bin\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.142990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142706 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-multus-cni-dir\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.142990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142756 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97178e0c-1c65-4e39-8b7d-09c74d097263-cnibin\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.142990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142766 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.142990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142786 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/97178e0c-1c65-4e39-8b7d-09c74d097263-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.142990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142813 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-dbus\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:45.142990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142840 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-systemd\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.142990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142874 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-socket-dir\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.142990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142876 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0076728-68c6-4f22-971f-68e8a7da5b56-host-slash\") pod \"iptables-alerter-krqrq\" (UID: \"a0076728-68c6-4f22-971f-68e8a7da5b56\") " pod="openshift-network-operator/iptables-alerter-krqrq" Aug 04 18:10:45.142990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142908 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-ovn-node-metrics-cert\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.142990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142935 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97178e0c-1c65-4e39-8b7d-09c74d097263-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.142990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142917 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-run-k8s-cni-cncf-io\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.142990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142964 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-cnibin\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.142990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.142990 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-multus-socket-dir-parent\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143059 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-run-netns\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143114 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-slash\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143126 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ceb0e538-c878-429c-b09b-bad693514c20-cni-binary-copy\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143137 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df338c86-7a60-48e5-ba0a-79a483d7275c-konnectivity-ca\") pod \"konnectivity-agent-845wp\" (UID: \"df338c86-7a60-48e5-ba0a-79a483d7275c\") " pod="kube-system/konnectivity-agent-845wp" Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143141 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-cni-netd\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143182 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-run-netns\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143220 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8d5v\" (UniqueName: \"kubernetes.io/projected/fb9a403b-8397-4699-bde5-4a19d861d697-kube-api-access-b8d5v\") pod \"node-resolver-rvt4h\" (UID: \"fb9a403b-8397-4699-bde5-4a19d861d697\") " pod="openshift-dns/node-resolver-rvt4h" Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143212 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-hostroot\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143247 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-multus-socket-dir-parent\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143292 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-cnibin\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143315 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs\") pod \"network-metrics-daemon-ntsjj\" (UID: \"11d07db5-806d-44e7-af32-be9ae461cbae\") " pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143355 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgjng\" (UniqueName: \"kubernetes.io/projected/ceb0e538-c878-429c-b09b-bad693514c20-kube-api-access-pgjng\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143426 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-kubernetes\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.143452 2526 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143562 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-node-log\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.143601 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs podName:11d07db5-806d-44e7-af32-be9ae461cbae nodeName:}" failed. No retries permitted until 2026-08-04 18:10:45.643578842 +0000 UTC m=+2.894861389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs") pod "network-metrics-daemon-ntsjj" (UID: "11d07db5-806d-44e7-af32-be9ae461cbae") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:45.143787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143625 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb9a403b-8397-4699-bde5-4a19d861d697-hosts-file\") pod \"node-resolver-rvt4h\" (UID: \"fb9a403b-8397-4699-bde5-4a19d861d697\") " pod="openshift-dns/node-resolver-rvt4h" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143654 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/884f5581-c681-4785-a065-fa174e052347-host\") pod \"node-ca-p9xc7\" (UID: \"884f5581-c681-4785-a065-fa174e052347\") " pod="openshift-image-registry/node-ca-p9xc7" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143695 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-modprobe-d\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143720 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtb9x\" (UniqueName: \"kubernetes.io/projected/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-kube-api-access-gtb9x\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143750 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb9a403b-8397-4699-bde5-4a19d861d697-tmp-dir\") pod \"node-resolver-rvt4h\" (UID: \"fb9a403b-8397-4699-bde5-4a19d861d697\") " pod="openshift-dns/node-resolver-rvt4h" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143780 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97178e0c-1c65-4e39-8b7d-09c74d097263-system-cni-dir\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143807 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddgl\" (UniqueName: \"kubernetes.io/projected/16ece713-3c41-4cdf-b03e-413a798e600f-kube-api-access-nddgl\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143828 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-run-netns\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143850 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143880 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-var-lib-cni-multus\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143906 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-sysctl-conf\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143929 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-host\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143957 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-log-socket\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.143997 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97178e0c-1c65-4e39-8b7d-09c74d097263-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144020 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-var-lib-cni-multus\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144024 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/97178e0c-1c65-4e39-8b7d-09c74d097263-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.144493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144061 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgrfh\" (UniqueName: \"kubernetes.io/projected/13c6d111-9af3-4928-a9fc-5533cae22fa4-kube-api-access-wgrfh\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144092 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-etc-kubernetes\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144147 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-ovnkube-script-lib\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144169 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txb6h\" (UniqueName: \"kubernetes.io/projected/97178e0c-1c65-4e39-8b7d-09c74d097263-kube-api-access-txb6h\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144185 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-etc-selinux\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144209 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-sys-fs\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144225 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-var-lib-kubelet\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144241 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0076728-68c6-4f22-971f-68e8a7da5b56-iptables-alerter-script\") pod \"iptables-alerter-krqrq\" (UID: \"a0076728-68c6-4f22-971f-68e8a7da5b56\") " pod="openshift-network-operator/iptables-alerter-krqrq" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144258 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-run-multus-certs\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144279 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/884f5581-c681-4785-a065-fa174e052347-serviceca\") pod \"node-ca-p9xc7\" (UID: \"884f5581-c681-4785-a065-fa174e052347\") " pod="openshift-image-registry/node-ca-p9xc7" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144306 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-lib-modules\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144331 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smj42\" (UniqueName: \"kubernetes.io/projected/a0076728-68c6-4f22-971f-68e8a7da5b56-kube-api-access-smj42\") pod \"iptables-alerter-krqrq\" (UID: \"a0076728-68c6-4f22-971f-68e8a7da5b56\") " pod="openshift-network-operator/iptables-alerter-krqrq" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144356 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-var-lib-openvswitch\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144378 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16ece713-3c41-4cdf-b03e-413a798e600f-tmp\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144376 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-sys-fs\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144396 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-etc-selinux\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144415 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-var-lib-kubelet\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.145258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144439 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-etc-kubernetes\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144447 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ceb0e538-c878-429c-b09b-bad693514c20-multus-daemon-config\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144442 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-host-run-multus-certs\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144476 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df338c86-7a60-48e5-ba0a-79a483d7275c-agent-certs\") pod \"konnectivity-agent-845wp\" (UID: \"df338c86-7a60-48e5-ba0a-79a483d7275c\") " pod="kube-system/konnectivity-agent-845wp" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144596 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-os-release\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144628 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk88z\" (UniqueName: \"kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z\") pod \"network-check-target-qmq4s\" (UID: \"0119f70f-109c-4ae9-b44e-2c3f69e61821\") " pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144712 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ceb0e538-c878-429c-b09b-bad693514c20-os-release\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144740 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-kubelet-config\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144756 2526 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144803 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-sysctl-d\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144826 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-run\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144852 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-registration-dir\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144878 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-device-dir\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144905 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vnsc\" (UniqueName: \"kubernetes.io/projected/11d07db5-806d-44e7-af32-be9ae461cbae-kube-api-access-4vnsc\") pod \"network-metrics-daemon-ntsjj\" (UID: \"11d07db5-806d-44e7-af32-be9ae461cbae\") " pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144930 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-run-systemd\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.144960 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.145009 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-env-overrides\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.146144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.145021 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-device-dir\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.146923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.145053 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ceb0e538-c878-429c-b09b-bad693514c20-multus-daemon-config\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.146923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.145073 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13c6d111-9af3-4928-a9fc-5533cae22fa4-registration-dir\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.146923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.145118 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:45.146923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.145145 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-sys\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.146923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.145195 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-var-lib-kubelet\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.146923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.145237 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16ece713-3c41-4cdf-b03e-413a798e600f-etc-tuned\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.146923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.145262 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-etc-openvswitch\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.146923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.145290 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-run-openvswitch\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.146923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.145317 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-cni-bin\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.146923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.145367 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97178e0c-1c65-4e39-8b7d-09c74d097263-os-release\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.149049 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.149026 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df338c86-7a60-48e5-ba0a-79a483d7275c-agent-certs\") pod \"konnectivity-agent-845wp\" (UID: \"df338c86-7a60-48e5-ba0a-79a483d7275c\") " pod="kube-system/konnectivity-agent-845wp" Aug 04 18:10:45.157126 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.157101 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgjng\" (UniqueName: \"kubernetes.io/projected/ceb0e538-c878-429c-b09b-bad693514c20-kube-api-access-pgjng\") pod \"multus-6m9tw\" (UID: \"ceb0e538-c878-429c-b09b-bad693514c20\") " pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.157863 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.157796 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgrfh\" (UniqueName: \"kubernetes.io/projected/13c6d111-9af3-4928-a9fc-5533cae22fa4-kube-api-access-wgrfh\") pod \"aws-ebs-csi-driver-node-vwjsm\" (UID: \"13c6d111-9af3-4928-a9fc-5533cae22fa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.159440 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.159419 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vnsc\" (UniqueName: \"kubernetes.io/projected/11d07db5-806d-44e7-af32-be9ae461cbae-kube-api-access-4vnsc\") pod \"network-metrics-daemon-ntsjj\" (UID: \"11d07db5-806d-44e7-af32-be9ae461cbae\") " pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:45.183801 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.183756 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal" event={"ID":"c07c5fa66b82b163d24a10ef1ee49cd1","Type":"ContainerStarted","Data":"8199018e6e2e789770e981cf0211a58e6cfe0d47802a81266fe6d77c07e4ccee"} Aug 04 18:10:45.185087 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.185061 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-165.ec2.internal" event={"ID":"a598d320185d24ca207d357f68b8dce3","Type":"ContainerStarted","Data":"370006406a23597eb438cd61e3bfe04363a447ac114b27c52ab57dd616fb27b8"} Aug 04 18:10:45.246599 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.246573 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-sysctl-conf\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.246741 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.246607 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-host\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.246741 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.246632 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-log-socket\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.246741 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.246710 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-host\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.246741 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.246734 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97178e0c-1c65-4e39-8b7d-09c74d097263-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.246949 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.246763 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/97178e0c-1c65-4e39-8b7d-09c74d097263-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.246949 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.246734 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-sysctl-conf\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.246949 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.246793 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-ovnkube-script-lib\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.246949 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.246776 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-log-socket\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.246949 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.246817 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txb6h\" (UniqueName: \"kubernetes.io/projected/97178e0c-1c65-4e39-8b7d-09c74d097263-kube-api-access-txb6h\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.246949 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.246844 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0076728-68c6-4f22-971f-68e8a7da5b56-iptables-alerter-script\") pod \"iptables-alerter-krqrq\" (UID: \"a0076728-68c6-4f22-971f-68e8a7da5b56\") " pod="openshift-network-operator/iptables-alerter-krqrq" Aug 04 18:10:45.246949 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.246903 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97178e0c-1c65-4e39-8b7d-09c74d097263-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.246949 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.246948 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/884f5581-c681-4785-a065-fa174e052347-serviceca\") pod \"node-ca-p9xc7\" (UID: \"884f5581-c681-4785-a065-fa174e052347\") " pod="openshift-image-registry/node-ca-p9xc7" Aug 04 18:10:45.247404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.246973 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-lib-modules\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.247404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247017 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smj42\" (UniqueName: \"kubernetes.io/projected/a0076728-68c6-4f22-971f-68e8a7da5b56-kube-api-access-smj42\") pod \"iptables-alerter-krqrq\" (UID: \"a0076728-68c6-4f22-971f-68e8a7da5b56\") " pod="openshift-network-operator/iptables-alerter-krqrq" Aug 04 18:10:45.247404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247043 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-var-lib-openvswitch\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.247404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247070 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16ece713-3c41-4cdf-b03e-413a798e600f-tmp\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.247404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247098 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk88z\" (UniqueName: \"kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z\") pod \"network-check-target-qmq4s\" (UID: \"0119f70f-109c-4ae9-b44e-2c3f69e61821\") " pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:10:45.247404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247124 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-kubelet-config\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:45.247404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247145 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-sysctl-d\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.247404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247164 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-run\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.247404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247190 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-run-systemd\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.247404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247219 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-lib-modules\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.247404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247265 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.247404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247308 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-var-lib-openvswitch\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247468 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/97178e0c-1c65-4e39-8b7d-09c74d097263-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247519 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-ovnkube-script-lib\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247661 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0076728-68c6-4f22-971f-68e8a7da5b56-iptables-alerter-script\") pod \"iptables-alerter-krqrq\" (UID: \"a0076728-68c6-4f22-971f-68e8a7da5b56\") " pod="openshift-network-operator/iptables-alerter-krqrq" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247671 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-sysctl-d\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247219 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247729 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-run\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247732 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-kubelet-config\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247741 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/884f5581-c681-4785-a065-fa174e052347-serviceca\") pod \"node-ca-p9xc7\" (UID: \"884f5581-c681-4785-a065-fa174e052347\") " pod="openshift-image-registry/node-ca-p9xc7" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247757 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-env-overrides\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247790 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247808 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-run-systemd\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247817 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-sys\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247843 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-var-lib-kubelet\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247867 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16ece713-3c41-4cdf-b03e-413a798e600f-etc-tuned\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247891 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-etc-openvswitch\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247900 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-sys\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.247993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247916 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-run-openvswitch\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247941 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-cni-bin\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247965 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97178e0c-1c65-4e39-8b7d-09c74d097263-os-release\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.247979 2526 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.247994 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9nv6\" (UniqueName: \"kubernetes.io/projected/884f5581-c681-4785-a065-fa174e052347-kube-api-access-f9nv6\") pod \"node-ca-p9xc7\" (UID: \"884f5581-c681-4785-a065-fa174e052347\") " pod="openshift-image-registry/node-ca-p9xc7" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248021 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-sysconfig\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248026 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-etc-openvswitch\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248067 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-systemd-units\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248097 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-run-ovn\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248100 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-var-lib-kubelet\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248126 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-kubelet\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248149 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-ovnkube-config\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248181 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97178e0c-1c65-4e39-8b7d-09c74d097263-cnibin\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248200 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-env-overrides\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248208 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/97178e0c-1c65-4e39-8b7d-09c74d097263-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248229 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-sysconfig\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248234 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-dbus\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248262 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-systemd\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.248873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248269 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-kubelet\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248286 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0076728-68c6-4f22-971f-68e8a7da5b56-host-slash\") pod \"iptables-alerter-krqrq\" (UID: \"a0076728-68c6-4f22-971f-68e8a7da5b56\") " pod="openshift-network-operator/iptables-alerter-krqrq" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248312 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-cni-bin\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248314 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-ovn-node-metrics-cert\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248353 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97178e0c-1c65-4e39-8b7d-09c74d097263-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248384 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-slash\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248409 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-cni-netd\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248435 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8d5v\" (UniqueName: \"kubernetes.io/projected/fb9a403b-8397-4699-bde5-4a19d861d697-kube-api-access-b8d5v\") pod \"node-resolver-rvt4h\" (UID: \"fb9a403b-8397-4699-bde5-4a19d861d697\") " pod="openshift-dns/node-resolver-rvt4h" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248476 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-kubernetes\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248481 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-dbus\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248500 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-node-log\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248528 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb9a403b-8397-4699-bde5-4a19d861d697-hosts-file\") pod \"node-resolver-rvt4h\" (UID: \"fb9a403b-8397-4699-bde5-4a19d861d697\") " pod="openshift-dns/node-resolver-rvt4h" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248536 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97178e0c-1c65-4e39-8b7d-09c74d097263-cnibin\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248552 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/884f5581-c681-4785-a065-fa174e052347-host\") pod \"node-ca-p9xc7\" (UID: \"884f5581-c681-4785-a065-fa174e052347\") " pod="openshift-image-registry/node-ca-p9xc7" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248577 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-modprobe-d\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248603 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtb9x\" (UniqueName: \"kubernetes.io/projected/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-kube-api-access-gtb9x\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248629 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb9a403b-8397-4699-bde5-4a19d861d697-tmp-dir\") pod \"node-resolver-rvt4h\" (UID: \"fb9a403b-8397-4699-bde5-4a19d861d697\") " pod="openshift-dns/node-resolver-rvt4h" Aug 04 18:10:45.249791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248657 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97178e0c-1c65-4e39-8b7d-09c74d097263-system-cni-dir\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248683 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddgl\" (UniqueName: \"kubernetes.io/projected/16ece713-3c41-4cdf-b03e-413a798e600f-kube-api-access-nddgl\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248710 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-run-netns\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248735 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248810 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248815 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-ovnkube-config\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.248844 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret podName:94ea3648-f325-45aa-b53b-f5c59bd1bd5a nodeName:}" failed. No retries permitted until 2026-08-04 18:10:45.748826516 +0000 UTC m=+3.000109051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret") pod "global-pull-secret-syncer-r59lx" (UID: "94ea3648-f325-45aa-b53b-f5c59bd1bd5a") : object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248885 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-systemd-units\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248890 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97178e0c-1c65-4e39-8b7d-09c74d097263-os-release\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248928 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-run-ovn\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248950 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-systemd\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248966 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-run-openvswitch\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248970 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0076728-68c6-4f22-971f-68e8a7da5b56-host-slash\") pod \"iptables-alerter-krqrq\" (UID: \"a0076728-68c6-4f22-971f-68e8a7da5b56\") " pod="openshift-network-operator/iptables-alerter-krqrq" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.248997 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/884f5581-c681-4785-a065-fa174e052347-host\") pod \"node-ca-p9xc7\" (UID: \"884f5581-c681-4785-a065-fa174e052347\") " pod="openshift-image-registry/node-ca-p9xc7" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.249005 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb9a403b-8397-4699-bde5-4a19d861d697-hosts-file\") pod \"node-resolver-rvt4h\" (UID: \"fb9a403b-8397-4699-bde5-4a19d861d697\") " pod="openshift-dns/node-resolver-rvt4h" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.249005 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/97178e0c-1c65-4e39-8b7d-09c74d097263-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.249057 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-cni-netd\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.250706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.249185 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-kubernetes\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.251599 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.249331 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16ece713-3c41-4cdf-b03e-413a798e600f-etc-modprobe-d\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.251599 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.249503 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-node-log\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.251599 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.249553 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97178e0c-1c65-4e39-8b7d-09c74d097263-system-cni-dir\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.251599 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.249608 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb9a403b-8397-4699-bde5-4a19d861d697-tmp-dir\") pod \"node-resolver-rvt4h\" (UID: \"fb9a403b-8397-4699-bde5-4a19d861d697\") " pod="openshift-dns/node-resolver-rvt4h" Aug 04 18:10:45.251599 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.249724 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-run-netns\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.251599 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.250105 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-host-slash\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.251599 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.250536 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16ece713-3c41-4cdf-b03e-413a798e600f-etc-tuned\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.251599 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.250589 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16ece713-3c41-4cdf-b03e-413a798e600f-tmp\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.251599 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.251566 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97178e0c-1c65-4e39-8b7d-09c74d097263-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.252423 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.252403 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-ovn-node-metrics-cert\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.258523 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.258504 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smj42\" (UniqueName: \"kubernetes.io/projected/a0076728-68c6-4f22-971f-68e8a7da5b56-kube-api-access-smj42\") pod \"iptables-alerter-krqrq\" (UID: \"a0076728-68c6-4f22-971f-68e8a7da5b56\") " pod="openshift-network-operator/iptables-alerter-krqrq" Aug 04 18:10:45.262504 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.262470 2526 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 18:10:45.262504 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.262506 2526 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 18:10:45.262635 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.262517 2526 projected.go:196] Error preparing data for projected volume kube-api-access-vk88z for pod openshift-network-diagnostics/network-check-target-qmq4s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:45.262635 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.262574 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z podName:0119f70f-109c-4ae9-b44e-2c3f69e61821 nodeName:}" failed. No retries permitted until 2026-08-04 18:10:45.762560618 +0000 UTC m=+3.013843159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vk88z" (UniqueName: "kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z") pod "network-check-target-qmq4s" (UID: "0119f70f-109c-4ae9-b44e-2c3f69e61821") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:45.268320 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.268292 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txb6h\" (UniqueName: \"kubernetes.io/projected/97178e0c-1c65-4e39-8b7d-09c74d097263-kube-api-access-txb6h\") pod \"multus-additional-cni-plugins-g2zhc\" (UID: \"97178e0c-1c65-4e39-8b7d-09c74d097263\") " pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.268422 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.268321 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8d5v\" (UniqueName: \"kubernetes.io/projected/fb9a403b-8397-4699-bde5-4a19d861d697-kube-api-access-b8d5v\") pod \"node-resolver-rvt4h\" (UID: \"fb9a403b-8397-4699-bde5-4a19d861d697\") " pod="openshift-dns/node-resolver-rvt4h" Aug 04 18:10:45.268649 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.268629 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddgl\" (UniqueName: \"kubernetes.io/projected/16ece713-3c41-4cdf-b03e-413a798e600f-kube-api-access-nddgl\") pod \"tuned-lp29p\" (UID: \"16ece713-3c41-4cdf-b03e-413a798e600f\") " pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.268755 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.268675 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtb9x\" (UniqueName: \"kubernetes.io/projected/5b7dd4cb-66b9-43fe-b1f7-998193ec0d16-kube-api-access-gtb9x\") pod \"ovnkube-node-7ns2w\" (UID: \"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.269719 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.269697 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9nv6\" (UniqueName: \"kubernetes.io/projected/884f5581-c681-4785-a065-fa174e052347-kube-api-access-f9nv6\") pod \"node-ca-p9xc7\" (UID: \"884f5581-c681-4785-a065-fa174e052347\") " pod="openshift-image-registry/node-ca-p9xc7" Aug 04 18:10:45.340541 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.340357 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-845wp" Aug 04 18:10:45.351854 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.351798 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" Aug 04 18:10:45.352739 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:45.352709 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf338c86_7a60_48e5_ba0a_79a483d7275c.slice/crio-7ccd01249cc173c5f35f43cf13a3b05a108060ca2aac3f3d9ce3ed73769b7e41 WatchSource:0}: Error finding container 7ccd01249cc173c5f35f43cf13a3b05a108060ca2aac3f3d9ce3ed73769b7e41: Status 404 returned error can't find the container with id 7ccd01249cc173c5f35f43cf13a3b05a108060ca2aac3f3d9ce3ed73769b7e41 Aug 04 18:10:45.361796 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:45.361769 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c6d111_9af3_4928_a9fc_5533cae22fa4.slice/crio-c93ec19ac319f27206d4b85e41cbead3ea3f0399d2efbcc9267f71f37e3ea479 WatchSource:0}: Error finding container c93ec19ac319f27206d4b85e41cbead3ea3f0399d2efbcc9267f71f37e3ea479: Status 404 returned error can't find the container with id c93ec19ac319f27206d4b85e41cbead3ea3f0399d2efbcc9267f71f37e3ea479 Aug 04 18:10:45.365927 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.365907 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6m9tw" Aug 04 18:10:45.374949 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.374927 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-krqrq" Aug 04 18:10:45.376225 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:45.376190 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb0e538_c878_429c_b09b_bad693514c20.slice/crio-931ab154e5b1f16caa1e5c843488d91ae051d69d8c870bbad9911adf9af1f409 WatchSource:0}: Error finding container 931ab154e5b1f16caa1e5c843488d91ae051d69d8c870bbad9911adf9af1f409: Status 404 returned error can't find the container with id 931ab154e5b1f16caa1e5c843488d91ae051d69d8c870bbad9911adf9af1f409 Aug 04 18:10:45.382194 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.382165 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:10:45.384661 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:45.384636 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0076728_68c6_4f22_971f_68e8a7da5b56.slice/crio-93bf25502b4117a011ded4b78913faa1305d19178cca51aeb5c53f406664aa9b WatchSource:0}: Error finding container 93bf25502b4117a011ded4b78913faa1305d19178cca51aeb5c53f406664aa9b: Status 404 returned error can't find the container with id 93bf25502b4117a011ded4b78913faa1305d19178cca51aeb5c53f406664aa9b Aug 04 18:10:45.390674 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.390649 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lp29p" Aug 04 18:10:45.392703 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:45.392668 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7dd4cb_66b9_43fe_b1f7_998193ec0d16.slice/crio-87a9d22a733856a1747aeb4ce9898865935bad619335873e383f968b2b69a848 WatchSource:0}: Error finding container 87a9d22a733856a1747aeb4ce9898865935bad619335873e383f968b2b69a848: Status 404 returned error can't find the container with id 87a9d22a733856a1747aeb4ce9898865935bad619335873e383f968b2b69a848 Aug 04 18:10:45.399212 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.399174 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rvt4h" Aug 04 18:10:45.401521 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:45.401495 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ece713_3c41_4cdf_b03e_413a798e600f.slice/crio-4ad80da7a6337c61f428b53eefcb41dc2176df2f3a873d037208d3d9e88eb9d0 WatchSource:0}: Error finding container 4ad80da7a6337c61f428b53eefcb41dc2176df2f3a873d037208d3d9e88eb9d0: Status 404 returned error can't find the container with id 4ad80da7a6337c61f428b53eefcb41dc2176df2f3a873d037208d3d9e88eb9d0 Aug 04 18:10:45.407421 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.407403 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g2zhc" Aug 04 18:10:45.408471 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:45.408446 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9a403b_8397_4699_bde5_4a19d861d697.slice/crio-70de2ea9da0d830cacd21d5e6a0b92603217cbb1039b54bc2453007f933912e3 WatchSource:0}: Error finding container 70de2ea9da0d830cacd21d5e6a0b92603217cbb1039b54bc2453007f933912e3: Status 404 returned error can't find the container with id 70de2ea9da0d830cacd21d5e6a0b92603217cbb1039b54bc2453007f933912e3 Aug 04 18:10:45.416060 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.415853 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p9xc7" Aug 04 18:10:45.417575 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:45.417555 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97178e0c_1c65_4e39_8b7d_09c74d097263.slice/crio-7dbfdb3d4bd9563f2ad9126f0253c4f5f26ebcc56e6f9c278cdc5dd7707307ea WatchSource:0}: Error finding container 7dbfdb3d4bd9563f2ad9126f0253c4f5f26ebcc56e6f9c278cdc5dd7707307ea: Status 404 returned error can't find the container with id 7dbfdb3d4bd9563f2ad9126f0253c4f5f26ebcc56e6f9c278cdc5dd7707307ea Aug 04 18:10:45.426547 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:10:45.426522 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884f5581_c681_4785_a065_fa174e052347.slice/crio-84229aa1f37cfdbd8e07f0ce2d6e9966b1e33b8d2d8f2dd670c719823c5b56ae WatchSource:0}: Error finding container 84229aa1f37cfdbd8e07f0ce2d6e9966b1e33b8d2d8f2dd670c719823c5b56ae: Status 404 returned error can't find the container with id 84229aa1f37cfdbd8e07f0ce2d6e9966b1e33b8d2d8f2dd670c719823c5b56ae Aug 04 18:10:45.652346 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.652265 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs\") pod \"network-metrics-daemon-ntsjj\" (UID: \"11d07db5-806d-44e7-af32-be9ae461cbae\") " pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:45.652718 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.652640 2526 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:45.652848 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.652728 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs podName:11d07db5-806d-44e7-af32-be9ae461cbae nodeName:}" failed. No retries permitted until 2026-08-04 18:10:46.652697995 +0000 UTC m=+3.903980540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs") pod "network-metrics-daemon-ntsjj" (UID: "11d07db5-806d-44e7-af32-be9ae461cbae") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:45.753455 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.753418 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:45.753616 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.753539 2526 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:45.753616 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.753596 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret podName:94ea3648-f325-45aa-b53b-f5c59bd1bd5a nodeName:}" failed. No retries permitted until 2026-08-04 18:10:46.753582056 +0000 UTC m=+4.004864582 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret") pod "global-pull-secret-syncer-r59lx" (UID: "94ea3648-f325-45aa-b53b-f5c59bd1bd5a") : object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:45.854089 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:45.854050 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk88z\" (UniqueName: \"kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z\") pod \"network-check-target-qmq4s\" (UID: \"0119f70f-109c-4ae9-b44e-2c3f69e61821\") " pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:10:45.854251 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.854186 2526 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 18:10:45.854251 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.854203 2526 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 18:10:45.854251 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.854213 2526 projected.go:196] Error preparing data for projected volume kube-api-access-vk88z for pod openshift-network-diagnostics/network-check-target-qmq4s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:45.854366 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:45.854273 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z podName:0119f70f-109c-4ae9-b44e-2c3f69e61821 nodeName:}" failed. No retries permitted until 2026-08-04 18:10:46.854258515 +0000 UTC m=+4.105541042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vk88z" (UniqueName: "kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z") pod "network-check-target-qmq4s" (UID: "0119f70f-109c-4ae9-b44e-2c3f69e61821") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:46.103594 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.103478 2526 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 18:05:45 +0000 UTC" deadline="2028-04-07 02:38:31.980500316 +0000 UTC" Aug 04 18:10:46.103594 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.103542 2526 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14672h27m45.876962753s" Aug 04 18:10:46.174666 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.174639 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:46.174805 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:46.174749 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:10:46.188525 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.188491 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2zhc" event={"ID":"97178e0c-1c65-4e39-8b7d-09c74d097263","Type":"ContainerStarted","Data":"7dbfdb3d4bd9563f2ad9126f0253c4f5f26ebcc56e6f9c278cdc5dd7707307ea"} Aug 04 18:10:46.189954 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.189901 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rvt4h" event={"ID":"fb9a403b-8397-4699-bde5-4a19d861d697","Type":"ContainerStarted","Data":"70de2ea9da0d830cacd21d5e6a0b92603217cbb1039b54bc2453007f933912e3"} Aug 04 18:10:46.191548 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.191511 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" event={"ID":"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16","Type":"ContainerStarted","Data":"87a9d22a733856a1747aeb4ce9898865935bad619335873e383f968b2b69a848"} Aug 04 18:10:46.192913 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.192863 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-krqrq" event={"ID":"a0076728-68c6-4f22-971f-68e8a7da5b56","Type":"ContainerStarted","Data":"93bf25502b4117a011ded4b78913faa1305d19178cca51aeb5c53f406664aa9b"} Aug 04 18:10:46.194322 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.194297 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6m9tw" event={"ID":"ceb0e538-c878-429c-b09b-bad693514c20","Type":"ContainerStarted","Data":"931ab154e5b1f16caa1e5c843488d91ae051d69d8c870bbad9911adf9af1f409"} Aug 04 18:10:46.195687 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.195615 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" event={"ID":"13c6d111-9af3-4928-a9fc-5533cae22fa4","Type":"ContainerStarted","Data":"c93ec19ac319f27206d4b85e41cbead3ea3f0399d2efbcc9267f71f37e3ea479"} Aug 04 18:10:46.196997 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.196973 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p9xc7" event={"ID":"884f5581-c681-4785-a065-fa174e052347","Type":"ContainerStarted","Data":"84229aa1f37cfdbd8e07f0ce2d6e9966b1e33b8d2d8f2dd670c719823c5b56ae"} Aug 04 18:10:46.198318 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.198287 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lp29p" event={"ID":"16ece713-3c41-4cdf-b03e-413a798e600f","Type":"ContainerStarted","Data":"4ad80da7a6337c61f428b53eefcb41dc2176df2f3a873d037208d3d9e88eb9d0"} Aug 04 18:10:46.199622 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.199602 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-845wp" event={"ID":"df338c86-7a60-48e5-ba0a-79a483d7275c","Type":"ContainerStarted","Data":"7ccd01249cc173c5f35f43cf13a3b05a108060ca2aac3f3d9ce3ed73769b7e41"} Aug 04 18:10:46.278774 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.278748 2526 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 18:10:46.661615 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.661519 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs\") pod \"network-metrics-daemon-ntsjj\" (UID: \"11d07db5-806d-44e7-af32-be9ae461cbae\") " pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:46.661787 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:46.661681 2526 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:46.661787 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:46.661758 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs podName:11d07db5-806d-44e7-af32-be9ae461cbae nodeName:}" failed. No retries permitted until 2026-08-04 18:10:48.661735938 +0000 UTC m=+5.913018470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs") pod "network-metrics-daemon-ntsjj" (UID: "11d07db5-806d-44e7-af32-be9ae461cbae") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:46.762982 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.762935 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:46.763203 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:46.763181 2526 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:46.763309 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:46.763255 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret podName:94ea3648-f325-45aa-b53b-f5c59bd1bd5a nodeName:}" failed. No retries permitted until 2026-08-04 18:10:48.763235083 +0000 UTC m=+6.014517617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret") pod "global-pull-secret-syncer-r59lx" (UID: "94ea3648-f325-45aa-b53b-f5c59bd1bd5a") : object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:46.863816 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:46.863785 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk88z\" (UniqueName: \"kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z\") pod \"network-check-target-qmq4s\" (UID: \"0119f70f-109c-4ae9-b44e-2c3f69e61821\") " pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:10:46.863963 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:46.863945 2526 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 18:10:46.864033 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:46.863974 2526 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 18:10:46.864033 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:46.863988 2526 projected.go:196] Error preparing data for projected volume kube-api-access-vk88z for pod openshift-network-diagnostics/network-check-target-qmq4s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:46.864168 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:46.864073 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z podName:0119f70f-109c-4ae9-b44e-2c3f69e61821 nodeName:}" failed. No retries permitted until 2026-08-04 18:10:48.864052235 +0000 UTC m=+6.115334784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vk88z" (UniqueName: "kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z") pod "network-check-target-qmq4s" (UID: "0119f70f-109c-4ae9-b44e-2c3f69e61821") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:47.104786 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:47.104403 2526 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 18:05:45 +0000 UTC" deadline="2028-05-12 12:22:39.301045754 +0000 UTC" Aug 04 18:10:47.104786 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:47.104679 2526 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15522h11m52.196376482s" Aug 04 18:10:47.177893 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:47.177181 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:47.177893 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:47.177323 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:10:47.177893 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:47.177733 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:10:47.177893 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:47.177841 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:10:47.207797 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:47.207737 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-165.ec2.internal" event={"ID":"a598d320185d24ca207d357f68b8dce3","Type":"ContainerStarted","Data":"afd87418b6998a0fa4ea21440c81dd0a4f281ba1db34cb7c89a466c873eacf05"} Aug 04 18:10:47.226460 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:47.226405 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-165.ec2.internal" podStartSLOduration=2.226385696 podStartE2EDuration="2.226385696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:47.22609738 +0000 UTC m=+4.477379933" watchObservedRunningTime="2026-08-04 18:10:47.226385696 +0000 UTC m=+4.477668248" Aug 04 18:10:48.174906 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:48.174872 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:48.175412 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:48.175023 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:10:48.229800 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:48.229765 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal" event={"ID":"c07c5fa66b82b163d24a10ef1ee49cd1","Type":"ContainerStarted","Data":"18a43e4c237c2f7de80a991dafcc154222192126f22029ed038745d31ce2df82"} Aug 04 18:10:48.681393 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:48.681355 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs\") pod \"network-metrics-daemon-ntsjj\" (UID: \"11d07db5-806d-44e7-af32-be9ae461cbae\") " pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:48.681593 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:48.681551 2526 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:48.681665 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:48.681618 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs podName:11d07db5-806d-44e7-af32-be9ae461cbae nodeName:}" failed. No retries permitted until 2026-08-04 18:10:52.681598086 +0000 UTC m=+9.932880615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs") pod "network-metrics-daemon-ntsjj" (UID: "11d07db5-806d-44e7-af32-be9ae461cbae") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:48.781791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:48.781748 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:48.781980 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:48.781905 2526 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:48.781980 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:48.781968 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret podName:94ea3648-f325-45aa-b53b-f5c59bd1bd5a nodeName:}" failed. No retries permitted until 2026-08-04 18:10:52.781951483 +0000 UTC m=+10.033234017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret") pod "global-pull-secret-syncer-r59lx" (UID: "94ea3648-f325-45aa-b53b-f5c59bd1bd5a") : object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:48.883357 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:48.882608 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk88z\" (UniqueName: \"kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z\") pod \"network-check-target-qmq4s\" (UID: \"0119f70f-109c-4ae9-b44e-2c3f69e61821\") " pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:10:48.883357 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:48.882834 2526 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 18:10:48.883357 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:48.882858 2526 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 18:10:48.883357 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:48.882871 2526 projected.go:196] Error preparing data for projected volume kube-api-access-vk88z for pod openshift-network-diagnostics/network-check-target-qmq4s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:48.883357 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:48.882928 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z podName:0119f70f-109c-4ae9-b44e-2c3f69e61821 nodeName:}" failed. No retries permitted until 2026-08-04 18:10:52.882911031 +0000 UTC m=+10.134193562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vk88z" (UniqueName: "kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z") pod "network-check-target-qmq4s" (UID: "0119f70f-109c-4ae9-b44e-2c3f69e61821") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:49.176771 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:49.176163 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:49.176771 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:49.176170 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:10:49.176771 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:49.176323 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:10:49.176771 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:49.176378 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:10:50.175540 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:50.175036 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:50.175540 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:50.175182 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:10:50.236822 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:50.236783 2526 generic.go:356] "Generic (PLEG): container finished" podID="c07c5fa66b82b163d24a10ef1ee49cd1" containerID="18a43e4c237c2f7de80a991dafcc154222192126f22029ed038745d31ce2df82" exitCode=0 Aug 04 18:10:50.237329 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:50.236839 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal" event={"ID":"c07c5fa66b82b163d24a10ef1ee49cd1","Type":"ContainerDied","Data":"18a43e4c237c2f7de80a991dafcc154222192126f22029ed038745d31ce2df82"} Aug 04 18:10:51.175706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:51.175677 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:51.175854 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:51.175728 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:10:51.175854 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:51.175800 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:10:51.176291 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:51.176245 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:10:52.174923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:52.174889 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:52.175442 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:52.175044 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:10:52.715143 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:52.715111 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs\") pod \"network-metrics-daemon-ntsjj\" (UID: \"11d07db5-806d-44e7-af32-be9ae461cbae\") " pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:52.715378 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:52.715247 2526 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:52.715378 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:52.715318 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs podName:11d07db5-806d-44e7-af32-be9ae461cbae nodeName:}" failed. No retries permitted until 2026-08-04 18:11:00.715296687 +0000 UTC m=+17.966579218 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs") pod "network-metrics-daemon-ntsjj" (UID: "11d07db5-806d-44e7-af32-be9ae461cbae") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:52.815569 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:52.815527 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:52.815761 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:52.815676 2526 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:52.815761 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:52.815754 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret podName:94ea3648-f325-45aa-b53b-f5c59bd1bd5a nodeName:}" failed. No retries permitted until 2026-08-04 18:11:00.815734064 +0000 UTC m=+18.067016597 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret") pod "global-pull-secret-syncer-r59lx" (UID: "94ea3648-f325-45aa-b53b-f5c59bd1bd5a") : object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:52.916430 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:52.916350 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk88z\" (UniqueName: \"kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z\") pod \"network-check-target-qmq4s\" (UID: \"0119f70f-109c-4ae9-b44e-2c3f69e61821\") " pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:10:52.916723 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:52.916531 2526 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 18:10:52.916723 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:52.916551 2526 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 18:10:52.916723 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:52.916563 2526 projected.go:196] Error preparing data for projected volume kube-api-access-vk88z for pod openshift-network-diagnostics/network-check-target-qmq4s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:52.916723 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:52.916621 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z podName:0119f70f-109c-4ae9-b44e-2c3f69e61821 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:00.91660167 +0000 UTC m=+18.167884194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vk88z" (UniqueName: "kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z") pod "network-check-target-qmq4s" (UID: "0119f70f-109c-4ae9-b44e-2c3f69e61821") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:53.175994 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:53.175914 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:53.176344 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:53.176062 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:10:53.176479 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:53.176457 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:10:53.176625 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:53.176553 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:10:54.175034 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:54.175000 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:54.175214 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:54.175136 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:10:55.175615 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:55.175531 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:55.176181 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:55.175673 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:10:55.176181 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:55.176002 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:10:55.176181 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:55.176083 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:10:56.175537 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:56.175478 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:56.175665 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:56.175573 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:10:56.250822 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:56.250788 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-845wp" event={"ID":"df338c86-7a60-48e5-ba0a-79a483d7275c","Type":"ContainerStarted","Data":"b9e21d50061896689db155033ab7bbf03008e404645b4442a95bd98dc3f8c847"} Aug 04 18:10:56.258398 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:56.258342 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal" event={"ID":"c07c5fa66b82b163d24a10ef1ee49cd1","Type":"ContainerStarted","Data":"76f971a62833709602f5fba5aac34329ce7f81c053ef6935b2c7d32c3d6b3133"} Aug 04 18:10:56.270493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:56.269988 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-845wp" podStartSLOduration=2.638104441 podStartE2EDuration="13.269954703s" totalImagesPullingTime="10.631850262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:56.269292251 +0000 UTC m=+13.520574799" watchObservedRunningTime="2026-08-04 18:10:56.269954703 +0000 UTC m=+13.521237251" Aug 04 18:10:56.287411 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:56.287365 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-165.ec2.internal" podStartSLOduration=11.287347963 podStartE2EDuration="11.287347963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:56.287195808 +0000 UTC m=+13.538478368" watchObservedRunningTime="2026-08-04 18:10:56.287347963 +0000 UTC m=+13.538630510" Aug 04 18:10:57.175505 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:57.175472 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:57.175652 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:57.175609 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:10:57.175711 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:57.175669 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:10:57.175812 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:57.175783 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:10:57.262410 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:57.262343 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" event={"ID":"13c6d111-9af3-4928-a9fc-5533cae22fa4","Type":"ContainerStarted","Data":"6274ab3b54e4642d38c9f2509fa4cd749a5e63d00abc813513d3bda5d03cc864"} Aug 04 18:10:57.264231 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:57.264204 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p9xc7" event={"ID":"884f5581-c681-4785-a065-fa174e052347","Type":"ContainerStarted","Data":"d1b52986fd2d0d5fa1253aff5f0bb0bea867cd5915d0f741a0f5bc1982bde3a8"} Aug 04 18:10:57.266323 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:57.266297 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lp29p" event={"ID":"16ece713-3c41-4cdf-b03e-413a798e600f","Type":"ContainerStarted","Data":"d79580de06ad4bd6f26a4fbc0935220855216828d0eb3fa2534184fbc364d1d8"} Aug 04 18:10:57.268682 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:57.268660 2526 generic.go:356] "Generic (PLEG): container finished" podID="97178e0c-1c65-4e39-8b7d-09c74d097263" containerID="4dcfe8f9b519942fb6afc0e8808c54d165c513436412b10cf41b3ca9135b715c" exitCode=0 Aug 04 18:10:57.268817 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:57.268756 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2zhc" event={"ID":"97178e0c-1c65-4e39-8b7d-09c74d097263","Type":"ContainerDied","Data":"4dcfe8f9b519942fb6afc0e8808c54d165c513436412b10cf41b3ca9135b715c"} Aug 04 18:10:57.275005 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:57.274978 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rvt4h" event={"ID":"fb9a403b-8397-4699-bde5-4a19d861d697","Type":"ContainerStarted","Data":"2877e7bd6c30b6c738c7cb158fb96dc048237cda10b41d25b7648b52390c0d67"} Aug 04 18:10:57.282764 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:57.282704 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p9xc7" podStartSLOduration=2.721565761 podStartE2EDuration="13.28268781s" totalImagesPullingTime="10.561122049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:57.282022249 +0000 UTC m=+14.533304796" watchObservedRunningTime="2026-08-04 18:10:57.28268781 +0000 UTC m=+14.533970358" Aug 04 18:10:57.304562 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:57.304480 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lp29p" podStartSLOduration=2.280907939 podStartE2EDuration="13.304464515s" totalImagesPullingTime="11.023556576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:57.304312811 +0000 UTC m=+14.555595359" watchObservedRunningTime="2026-08-04 18:10:57.304464515 +0000 UTC m=+14.555747066" Aug 04 18:10:57.321536 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:57.321499 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rvt4h" podStartSLOduration=2.743925769 podStartE2EDuration="13.321488966s" totalImagesPullingTime="10.577563197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:57.321130156 +0000 UTC m=+14.572412704" watchObservedRunningTime="2026-08-04 18:10:57.321488966 +0000 UTC m=+14.572771512" Aug 04 18:10:58.175076 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:58.175039 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:10:58.175216 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:58.175173 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:10:58.278781 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:58.278743 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-krqrq" event={"ID":"a0076728-68c6-4f22-971f-68e8a7da5b56","Type":"ContainerStarted","Data":"287ba153ec2f4ccf4025466e25a8af6b4be897fb806c5deef845e54608ee4178"} Aug 04 18:10:58.297952 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:58.297909 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-krqrq" podStartSLOduration=4.698486667 podStartE2EDuration="15.297895008s" totalImagesPullingTime="10.599408341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:58.29729731 +0000 UTC m=+15.548579872" watchObservedRunningTime="2026-08-04 18:10:58.297895008 +0000 UTC m=+15.549177554" Aug 04 18:10:59.027754 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:59.027718 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-845wp" Aug 04 18:10:59.028385 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:59.028362 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-845wp" Aug 04 18:10:59.174703 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:59.174667 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:10:59.174876 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:59.174816 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:10:59.174876 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:10:59.174867 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:10:59.175005 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:10:59.174990 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:11:00.175318 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:00.175284 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:00.176156 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:00.175413 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:11:00.777116 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:00.777078 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs\") pod \"network-metrics-daemon-ntsjj\" (UID: \"11d07db5-806d-44e7-af32-be9ae461cbae\") " pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:00.777302 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:00.777238 2526 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:11:00.777357 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:00.777313 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs podName:11d07db5-806d-44e7-af32-be9ae461cbae nodeName:}" failed. No retries permitted until 2026-08-04 18:11:16.777290678 +0000 UTC m=+34.028573206 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs") pod "network-metrics-daemon-ntsjj" (UID: "11d07db5-806d-44e7-af32-be9ae461cbae") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:11:00.877716 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:00.877677 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:00.877865 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:00.877844 2526 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 18:11:00.877918 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:00.877909 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret podName:94ea3648-f325-45aa-b53b-f5c59bd1bd5a nodeName:}" failed. No retries permitted until 2026-08-04 18:11:16.877894196 +0000 UTC m=+34.129176726 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret") pod "global-pull-secret-syncer-r59lx" (UID: "94ea3648-f325-45aa-b53b-f5c59bd1bd5a") : object "kube-system"/"original-pull-secret" not registered Aug 04 18:11:00.978798 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:00.978764 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk88z\" (UniqueName: \"kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z\") pod \"network-check-target-qmq4s\" (UID: \"0119f70f-109c-4ae9-b44e-2c3f69e61821\") " pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:00.978951 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:00.978915 2526 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 18:11:00.978951 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:00.978936 2526 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 18:11:00.978951 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:00.978949 2526 projected.go:196] Error preparing data for projected volume kube-api-access-vk88z for pod openshift-network-diagnostics/network-check-target-qmq4s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:11:00.979095 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:00.979018 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z podName:0119f70f-109c-4ae9-b44e-2c3f69e61821 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:16.97900376 +0000 UTC m=+34.230286285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vk88z" (UniqueName: "kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z") pod "network-check-target-qmq4s" (UID: "0119f70f-109c-4ae9-b44e-2c3f69e61821") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:11:01.176097 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:01.175436 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:01.176097 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:01.175595 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:11:01.176097 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:01.175913 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:01.176097 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:01.176015 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:11:02.175484 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:02.175452 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:02.175645 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:02.175580 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:11:02.692716 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:02.692683 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-845wp" Aug 04 18:11:02.693159 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:02.692813 2526 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 04 18:11:02.694155 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:02.694118 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-845wp" Aug 04 18:11:03.175610 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:03.175532 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:03.175769 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:03.175638 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:11:03.175769 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:03.175692 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:03.175880 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:03.175814 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:11:04.174742 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:04.174706 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:04.175233 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:04.174814 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:11:05.174863 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:05.174824 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:05.174863 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:05.174858 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:05.175493 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:05.174953 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:11:05.175493 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:05.175035 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:11:06.174996 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:06.174949 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:06.175546 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:06.175088 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:11:07.175686 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:07.175489 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:07.176123 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:07.175498 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:07.176123 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:07.175800 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:11:07.176123 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:07.175866 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:11:08.043440 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:08.043293 2526 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 18:11:08.099290 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:08.099036 2526 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T18:11:08.043313105Z","UUID":"9ea46bf6-729b-4e64-b2ba-c4158b667143","Handler":null,"Name":"","Endpoint":""} Aug 04 18:11:08.100511 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:08.100475 2526 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 18:11:08.100591 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:08.100518 2526 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 18:11:08.174861 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:08.174836 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:08.174949 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:08.174934 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:11:08.299121 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:08.299098 2526 generic.go:356] "Generic (PLEG): container finished" podID="97178e0c-1c65-4e39-8b7d-09c74d097263" containerID="98da931c69f4609ce147df1146562ebf6be613ded5cfef3fac8afd3ac192c0da" exitCode=0 Aug 04 18:11:08.299419 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:08.299165 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2zhc" event={"ID":"97178e0c-1c65-4e39-8b7d-09c74d097263","Type":"ContainerDied","Data":"98da931c69f4609ce147df1146562ebf6be613ded5cfef3fac8afd3ac192c0da"} Aug 04 18:11:08.301362 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:08.301330 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" event={"ID":"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16","Type":"ContainerStarted","Data":"c307ceecc4ff5f4c4e4e04c88449cc09f99f099bf46c3da8bb69430e4928ea1d"} Aug 04 18:11:08.301481 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:08.301394 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" event={"ID":"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16","Type":"ContainerStarted","Data":"6d40aa0a17d15bb6aa4fcd1a66f8140be26c781bd3a3367d1cf25ccefaf51bc8"} Aug 04 18:11:08.301481 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:08.301439 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" event={"ID":"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16","Type":"ContainerStarted","Data":"4f1d493381a41293261233fc1fb2f627a001164781eeea701ff60800cc2b3674"} Aug 04 18:11:08.302910 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:08.302889 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6m9tw" event={"ID":"ceb0e538-c878-429c-b09b-bad693514c20","Type":"ContainerStarted","Data":"08e02fedf4cef5c3d17611dba7978a5a740135f90f0c2e9ecdc78992bfe3e2f2"} Aug 04 18:11:08.304765 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:08.304745 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" event={"ID":"13c6d111-9af3-4928-a9fc-5533cae22fa4","Type":"ContainerStarted","Data":"64f9c19bc4b4d044a4e9248a8e06a2283a3bd25dce50f62aca5b17d5c2558232"} Aug 04 18:11:08.344553 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:08.344510 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6m9tw" podStartSLOduration=2.805103155 podStartE2EDuration="25.344494031s" totalImagesPullingTime="22.539390876s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:08.34398026 +0000 UTC m=+25.595262806" watchObservedRunningTime="2026-08-04 18:11:08.344494031 +0000 UTC m=+25.595776630" Aug 04 18:11:09.175557 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:09.175527 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:09.175749 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:09.175667 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:11:09.175749 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:09.175712 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:09.175858 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:09.175795 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:11:09.312411 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:09.312323 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" event={"ID":"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16","Type":"ContainerStarted","Data":"917c87500f94c43420693ac528bd08600e5e920533c027e65ee1aea803abeadb"} Aug 04 18:11:09.312957 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:09.312420 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" event={"ID":"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16","Type":"ContainerStarted","Data":"d0349eae62672f849fea5ca635a7409bbc54a5d009f782502281743dd76f2fa7"} Aug 04 18:11:09.312957 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:09.312469 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" event={"ID":"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16","Type":"ContainerStarted","Data":"caa017270ece5184c0969698db879c3ac85c2b047d9d5b53d1dfd39d40f6223f"} Aug 04 18:11:10.175589 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:10.175513 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:10.175789 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:10.175607 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:11:10.315839 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:10.315805 2526 generic.go:356] "Generic (PLEG): container finished" podID="97178e0c-1c65-4e39-8b7d-09c74d097263" containerID="a1a612f41554ee8791246ab604f76d3d8aca0f317ff37811b7f9c965dd35af66" exitCode=0 Aug 04 18:11:10.316397 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:10.315844 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2zhc" event={"ID":"97178e0c-1c65-4e39-8b7d-09c74d097263","Type":"ContainerDied","Data":"a1a612f41554ee8791246ab604f76d3d8aca0f317ff37811b7f9c965dd35af66"} Aug 04 18:11:10.317878 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:10.317855 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" event={"ID":"13c6d111-9af3-4928-a9fc-5533cae22fa4","Type":"ContainerStarted","Data":"3e14facc93ee4ef427368d5807df683a296292b76bd06aaa8e03e1116ce2f90a"} Aug 04 18:11:10.398845 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:10.398809 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwjsm" podStartSLOduration=3.640731347 podStartE2EDuration="27.398796366s" totalImagesPullingTime="23.758065019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:10.398223701 +0000 UTC m=+27.649506247" watchObservedRunningTime="2026-08-04 18:11:10.398796366 +0000 UTC m=+27.650078913" Aug 04 18:11:10.537984 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:10.537909 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:10.664183 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:10.664157 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:11.175348 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:11.175324 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:11.175487 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:11.175427 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:11:11.175535 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:11.175486 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:11.175585 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:11.175568 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:11:11.322789 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:11.322713 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" event={"ID":"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16","Type":"ContainerStarted","Data":"64459705eb2e387cd6139ba3ecc6af1f415664b3b971ac23e63ce19415781de6"} Aug 04 18:11:12.174602 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:12.174570 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:12.174746 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:12.174669 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:11:12.326772 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:12.326744 2526 generic.go:356] "Generic (PLEG): container finished" podID="97178e0c-1c65-4e39-8b7d-09c74d097263" containerID="df9a90e6a710996698c1a1a7c2e226d424e7e147dce47066d587e70d328b4117" exitCode=0 Aug 04 18:11:12.327239 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:12.326800 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2zhc" event={"ID":"97178e0c-1c65-4e39-8b7d-09c74d097263","Type":"ContainerDied","Data":"df9a90e6a710996698c1a1a7c2e226d424e7e147dce47066d587e70d328b4117"} Aug 04 18:11:13.175836 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:13.175573 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:13.176011 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:13.175658 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:13.176011 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:13.175866 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:11:13.176011 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:13.175943 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:11:13.333473 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:13.333443 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" event={"ID":"5b7dd4cb-66b9-43fe-b1f7-998193ec0d16","Type":"ContainerStarted","Data":"f5af45b511a17b08ad676f0a0782df13dd6aaaef872ea4f72c359801fae05bbb"} Aug 04 18:11:13.334042 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:13.333791 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:11:13.334042 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:13.333856 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:11:13.334042 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:13.333904 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:11:13.353397 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:13.353373 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:11:13.353519 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:13.353478 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:11:13.365641 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:13.365595 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" podStartSLOduration=6.851903875 podStartE2EDuration="29.365584161s" totalImagesPullingTime="22.513680286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:13.364478057 +0000 UTC m=+30.615760603" watchObservedRunningTime="2026-08-04 18:11:13.365584161 +0000 UTC m=+30.616866708" Aug 04 18:11:13.378312 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:13.378288 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:13.751981 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:13.751932 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:13.817478 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:13.817426 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:14.175580 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:14.175509 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:14.175723 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:14.175638 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:11:14.977717 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:14.977680 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r59lx"] Aug 04 18:11:14.978167 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:14.977832 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:14.978167 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:14.977934 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:11:14.980940 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:14.980903 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ntsjj"] Aug 04 18:11:14.981071 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:14.981036 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:14.981155 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:14.981138 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:11:14.981843 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:14.981682 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qmq4s"] Aug 04 18:11:14.981843 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:14.981774 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:14.982028 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:14.981852 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:11:14.982989 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:14.982861 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:15.083781 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:15.083696 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:16.174862 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:16.174822 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:16.174862 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:16.174846 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:16.175369 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:16.174828 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:16.175369 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:16.174945 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:11:16.175369 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:16.175061 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:11:16.175369 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:16.175162 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:11:16.796928 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:16.796764 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs\") pod \"network-metrics-daemon-ntsjj\" (UID: \"11d07db5-806d-44e7-af32-be9ae461cbae\") " pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:16.797088 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:16.796907 2526 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:11:16.797088 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:16.797043 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs podName:11d07db5-806d-44e7-af32-be9ae461cbae nodeName:}" failed. No retries permitted until 2026-08-04 18:11:48.797026118 +0000 UTC m=+66.048308643 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs") pod "network-metrics-daemon-ntsjj" (UID: "11d07db5-806d-44e7-af32-be9ae461cbae") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:11:16.897553 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:16.897526 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:16.897672 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:16.897638 2526 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 18:11:16.897722 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:16.897710 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret podName:94ea3648-f325-45aa-b53b-f5c59bd1bd5a nodeName:}" failed. No retries permitted until 2026-08-04 18:11:48.897691513 +0000 UTC m=+66.148974044 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret") pod "global-pull-secret-syncer-r59lx" (UID: "94ea3648-f325-45aa-b53b-f5c59bd1bd5a") : object "kube-system"/"original-pull-secret" not registered Aug 04 18:11:16.998496 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:16.998472 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk88z\" (UniqueName: \"kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z\") pod \"network-check-target-qmq4s\" (UID: \"0119f70f-109c-4ae9-b44e-2c3f69e61821\") " pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:16.998622 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:16.998596 2526 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 18:11:16.998622 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:16.998611 2526 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 18:11:16.998622 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:16.998619 2526 projected.go:196] Error preparing data for projected volume kube-api-access-vk88z for pod openshift-network-diagnostics/network-check-target-qmq4s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:11:16.998715 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:16.998666 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z podName:0119f70f-109c-4ae9-b44e-2c3f69e61821 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:48.99865057 +0000 UTC m=+66.249933099 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vk88z" (UniqueName: "kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z") pod "network-check-target-qmq4s" (UID: "0119f70f-109c-4ae9-b44e-2c3f69e61821") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:11:17.345890 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:17.345858 2526 generic.go:356] "Generic (PLEG): container finished" podID="97178e0c-1c65-4e39-8b7d-09c74d097263" containerID="2e0324e57045a121ab9aa34ace8927bc1b28a08afdf34d6327000be701dfc1c7" exitCode=0 Aug 04 18:11:17.346256 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:17.345912 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2zhc" event={"ID":"97178e0c-1c65-4e39-8b7d-09c74d097263","Type":"ContainerDied","Data":"2e0324e57045a121ab9aa34ace8927bc1b28a08afdf34d6327000be701dfc1c7"} Aug 04 18:11:18.175035 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:18.175003 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:18.175224 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:18.175004 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:18.175224 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:18.175105 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:11:18.175224 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:18.175012 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:18.175224 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:18.175187 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:11:18.175392 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:18.175289 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:11:18.351288 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:18.351258 2526 generic.go:356] "Generic (PLEG): container finished" podID="97178e0c-1c65-4e39-8b7d-09c74d097263" containerID="69753b5659187621d068cdde862386def4045a72e5d45adf469b81af77eb1271" exitCode=0 Aug 04 18:11:18.351635 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:18.351319 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2zhc" event={"ID":"97178e0c-1c65-4e39-8b7d-09c74d097263","Type":"ContainerDied","Data":"69753b5659187621d068cdde862386def4045a72e5d45adf469b81af77eb1271"} Aug 04 18:11:19.356710 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:19.356670 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2zhc" event={"ID":"97178e0c-1c65-4e39-8b7d-09c74d097263","Type":"ContainerStarted","Data":"85d9f0f81ad28c2fd12406923b71f03f9f391344c00371ee860d6a4fa7b1551c"} Aug 04 18:11:19.385184 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:19.385073 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g2zhc" podStartSLOduration=7.476985305 podStartE2EDuration="35.385056835s" totalImagesPullingTime="27.90807153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:19.384673451 +0000 UTC m=+36.635955998" watchObservedRunningTime="2026-08-04 18:11:19.385056835 +0000 UTC m=+36.636339382" Aug 04 18:11:19.512753 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:19.512727 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:19.605158 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:19.605132 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:20.175315 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.175287 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:20.175469 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.175381 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:20.175469 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:20.175389 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r59lx" podUID="94ea3648-f325-45aa-b53b-f5c59bd1bd5a" Aug 04 18:11:20.175583 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:20.175473 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntsjj" podUID="11d07db5-806d-44e7-af32-be9ae461cbae" Aug 04 18:11:20.175583 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.175506 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:20.175583 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:20.175562 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmq4s" podUID="0119f70f-109c-4ae9-b44e-2c3f69e61821" Aug 04 18:11:20.604553 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.604498 2526 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-165.ec2.internal" event="NodeReady" Aug 04 18:11:20.604864 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.604578 2526 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 18:11:20.660715 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.660693 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f4c467488-kwzr5"] Aug 04 18:11:20.692639 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.692617 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kdk4v"] Aug 04 18:11:20.692751 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.692736 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.700265 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.700232 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 04 18:11:20.700777 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.700727 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 04 18:11:20.700872 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.700822 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 04 18:11:20.701095 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.701077 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zjg4n\"" Aug 04 18:11:20.708416 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.708398 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 04 18:11:20.712459 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.712444 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xr2hn"] Aug 04 18:11:20.712578 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.712561 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:20.715908 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.715889 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 18:11:20.716051 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.715975 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-khq2z\"" Aug 04 18:11:20.716146 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.716123 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 18:11:20.724505 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.724489 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f4c467488-kwzr5"] Aug 04 18:11:20.724592 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.724513 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xr2hn"] Aug 04 18:11:20.724592 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.724526 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kdk4v"] Aug 04 18:11:20.724746 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.724602 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xr2hn" Aug 04 18:11:20.729232 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.729216 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rwz87\"" Aug 04 18:11:20.729366 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.729349 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 18:11:20.729615 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.729604 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 18:11:20.730010 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.729991 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 18:11:20.826512 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.826491 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4pbv\" (UniqueName: \"kubernetes.io/projected/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-kube-api-access-c4pbv\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:20.826610 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.826522 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e375a34f-0263-4345-9317-067d37d3a95c-ca-trust-extracted\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.826610 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.826540 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-config-volume\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:20.826610 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.826562 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-tmp-dir\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:20.826610 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.826597 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e375a34f-0263-4345-9317-067d37d3a95c-image-registry-private-configuration\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.826743 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.826690 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:20.826743 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.826709 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9e9f16d-731b-4afb-95be-304a641d3e4d-cert\") pod \"ingress-canary-xr2hn\" (UID: \"c9e9f16d-731b-4afb-95be-304a641d3e4d\") " pod="openshift-ingress-canary/ingress-canary-xr2hn" Aug 04 18:11:20.826743 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.826725 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e375a34f-0263-4345-9317-067d37d3a95c-registry-certificates\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.826830 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.826767 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-bound-sa-token\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.826830 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.826792 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e375a34f-0263-4345-9317-067d37d3a95c-installation-pull-secrets\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.826830 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.826812 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb56r\" (UniqueName: \"kubernetes.io/projected/c9e9f16d-731b-4afb-95be-304a641d3e4d-kube-api-access-cb56r\") pod \"ingress-canary-xr2hn\" (UID: \"c9e9f16d-731b-4afb-95be-304a641d3e4d\") " pod="openshift-ingress-canary/ingress-canary-xr2hn" Aug 04 18:11:20.826943 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.826835 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsptg\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-kube-api-access-qsptg\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.826943 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.826867 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.826943 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.826881 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e375a34f-0263-4345-9317-067d37d3a95c-trusted-ca\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.927815 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.927772 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-tmp-dir\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:20.927815 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.927797 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e375a34f-0263-4345-9317-067d37d3a95c-image-registry-private-configuration\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.927928 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.927829 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:20.927928 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.927847 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9e9f16d-731b-4afb-95be-304a641d3e4d-cert\") pod \"ingress-canary-xr2hn\" (UID: \"c9e9f16d-731b-4afb-95be-304a641d3e4d\") " pod="openshift-ingress-canary/ingress-canary-xr2hn" Aug 04 18:11:20.927928 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.927861 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e375a34f-0263-4345-9317-067d37d3a95c-registry-certificates\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.928101 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.928009 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-bound-sa-token\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.928101 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.928040 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e375a34f-0263-4345-9317-067d37d3a95c-installation-pull-secrets\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.928101 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.928060 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb56r\" (UniqueName: \"kubernetes.io/projected/c9e9f16d-731b-4afb-95be-304a641d3e4d-kube-api-access-cb56r\") pod \"ingress-canary-xr2hn\" (UID: \"c9e9f16d-731b-4afb-95be-304a641d3e4d\") " pod="openshift-ingress-canary/ingress-canary-xr2hn" Aug 04 18:11:20.928101 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.928080 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsptg\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-kube-api-access-qsptg\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.928272 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.928124 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.928272 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.928157 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e375a34f-0263-4345-9317-067d37d3a95c-trusted-ca\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.928272 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.928199 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4pbv\" (UniqueName: \"kubernetes.io/projected/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-kube-api-access-c4pbv\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:20.928272 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.928214 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-tmp-dir\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:20.928272 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.928223 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e375a34f-0263-4345-9317-067d37d3a95c-ca-trust-extracted\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.928272 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.928249 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-config-volume\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:20.928515 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:20.928303 2526 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 18:11:20.928515 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:20.928318 2526 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4c467488-kwzr5: secret "image-registry-tls" not found Aug 04 18:11:20.928515 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:20.928343 2526 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 18:11:20.928515 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:20.928369 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls podName:e375a34f-0263-4345-9317-067d37d3a95c nodeName:}" failed. No retries permitted until 2026-08-04 18:11:21.428351161 +0000 UTC m=+38.679633700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls") pod "image-registry-7f4c467488-kwzr5" (UID: "e375a34f-0263-4345-9317-067d37d3a95c") : secret "image-registry-tls" not found Aug 04 18:11:20.928515 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:20.928389 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls podName:7439e0a9-d4de-4b6a-ae78-48a889eff5a1 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:21.428378145 +0000 UTC m=+38.679660689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls") pod "dns-default-kdk4v" (UID: "7439e0a9-d4de-4b6a-ae78-48a889eff5a1") : secret "dns-default-metrics-tls" not found Aug 04 18:11:20.928824 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.928516 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e375a34f-0263-4345-9317-067d37d3a95c-registry-certificates\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.928910 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.928889 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-config-volume\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:20.928910 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.928905 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e375a34f-0263-4345-9317-067d37d3a95c-ca-trust-extracted\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.929240 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.929218 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e375a34f-0263-4345-9317-067d37d3a95c-trusted-ca\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.932059 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.932039 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e375a34f-0263-4345-9317-067d37d3a95c-installation-pull-secrets\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.932197 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.932179 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e375a34f-0263-4345-9317-067d37d3a95c-image-registry-private-configuration\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.932240 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.932225 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9e9f16d-731b-4afb-95be-304a641d3e4d-cert\") pod \"ingress-canary-xr2hn\" (UID: \"c9e9f16d-731b-4afb-95be-304a641d3e4d\") " pod="openshift-ingress-canary/ingress-canary-xr2hn" Aug 04 18:11:20.942344 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.942319 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsptg\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-kube-api-access-qsptg\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.943200 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.943180 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-bound-sa-token\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:20.943900 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.943880 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4pbv\" (UniqueName: \"kubernetes.io/projected/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-kube-api-access-c4pbv\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:20.944903 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:20.944887 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb56r\" (UniqueName: \"kubernetes.io/projected/c9e9f16d-731b-4afb-95be-304a641d3e4d-kube-api-access-cb56r\") pod \"ingress-canary-xr2hn\" (UID: \"c9e9f16d-731b-4afb-95be-304a641d3e4d\") " pod="openshift-ingress-canary/ingress-canary-xr2hn" Aug 04 18:11:21.032420 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:21.032397 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xr2hn" Aug 04 18:11:21.246324 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:21.246146 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xr2hn"] Aug 04 18:11:21.250238 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:11:21.250214 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e9f16d_731b_4afb_95be_304a641d3e4d.slice/crio-7eb91f42829eb3e40694dfdb8e14347ac85cabe8248946f83b2adca58b75d625 WatchSource:0}: Error finding container 7eb91f42829eb3e40694dfdb8e14347ac85cabe8248946f83b2adca58b75d625: Status 404 returned error can't find the container with id 7eb91f42829eb3e40694dfdb8e14347ac85cabe8248946f83b2adca58b75d625 Aug 04 18:11:21.364460 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:21.364427 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xr2hn" event={"ID":"c9e9f16d-731b-4afb-95be-304a641d3e4d","Type":"ContainerStarted","Data":"7eb91f42829eb3e40694dfdb8e14347ac85cabe8248946f83b2adca58b75d625"} Aug 04 18:11:21.432882 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:21.432814 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:21.432882 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:21.432867 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:21.433046 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:21.432991 2526 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 18:11:21.433081 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:21.433047 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls podName:7439e0a9-d4de-4b6a-ae78-48a889eff5a1 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:22.433034606 +0000 UTC m=+39.684317131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls") pod "dns-default-kdk4v" (UID: "7439e0a9-d4de-4b6a-ae78-48a889eff5a1") : secret "dns-default-metrics-tls" not found Aug 04 18:11:21.433117 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:21.432995 2526 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 18:11:21.433117 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:21.433094 2526 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4c467488-kwzr5: secret "image-registry-tls" not found Aug 04 18:11:21.433176 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:21.433132 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls podName:e375a34f-0263-4345-9317-067d37d3a95c nodeName:}" failed. No retries permitted until 2026-08-04 18:11:22.433122059 +0000 UTC m=+39.684404601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls") pod "image-registry-7f4c467488-kwzr5" (UID: "e375a34f-0263-4345-9317-067d37d3a95c") : secret "image-registry-tls" not found Aug 04 18:11:21.740257 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:21.740175 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:21.834903 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:21.834870 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:22.175736 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.175648 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:22.175736 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.175678 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:22.175936 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.175648 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:22.182447 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.182426 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vzv57"] Aug 04 18:11:22.184264 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.184246 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 18:11:22.184575 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.184550 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 18:11:22.184681 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.184635 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wz6f8\"" Aug 04 18:11:22.184681 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.184648 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 18:11:22.184792 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.184692 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kp5sr\"" Aug 04 18:11:22.184792 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.184637 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 18:11:22.207158 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.207138 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vzv57"] Aug 04 18:11:22.207267 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.207257 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.216216 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.216197 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 18:11:22.216296 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.216231 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 18:11:22.216296 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.216231 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 18:11:22.229230 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.229204 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:22.230522 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.230502 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 18:11:22.231781 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.231759 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r6h4z\"" Aug 04 18:11:22.341024 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.340996 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83ceb079-6106-4ef2-a063-3506b4eb60a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.341155 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.341032 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.341155 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.341106 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83ceb079-6106-4ef2-a063-3506b4eb60a2-data-volume\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.341155 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.341148 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfkmd\" (UniqueName: \"kubernetes.io/projected/83ceb079-6106-4ef2-a063-3506b4eb60a2-kube-api-access-mfkmd\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.341303 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.341250 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83ceb079-6106-4ef2-a063-3506b4eb60a2-crio-socket\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.370389 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.370353 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:22.442251 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.442178 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfkmd\" (UniqueName: \"kubernetes.io/projected/83ceb079-6106-4ef2-a063-3506b4eb60a2-kube-api-access-mfkmd\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.442251 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.442223 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:22.442436 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.442269 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83ceb079-6106-4ef2-a063-3506b4eb60a2-crio-socket\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.442436 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.442303 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83ceb079-6106-4ef2-a063-3506b4eb60a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.442436 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.442330 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.442436 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.442366 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:22.442436 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:22.442377 2526 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 18:11:22.442436 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.442391 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83ceb079-6106-4ef2-a063-3506b4eb60a2-data-volume\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.442436 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:22.442398 2526 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4c467488-kwzr5: secret "image-registry-tls" not found Aug 04 18:11:22.442436 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:22.442418 2526 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 18:11:22.442676 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:22.442451 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls podName:e375a34f-0263-4345-9317-067d37d3a95c nodeName:}" failed. No retries permitted until 2026-08-04 18:11:24.442434617 +0000 UTC m=+41.693717154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls") pod "image-registry-7f4c467488-kwzr5" (UID: "e375a34f-0263-4345-9317-067d37d3a95c") : secret "image-registry-tls" not found Aug 04 18:11:22.442676 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:22.442480 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83ceb079-6106-4ef2-a063-3506b4eb60a2-kube-rbac-proxy-cm podName:83ceb079-6106-4ef2-a063-3506b4eb60a2 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:22.942464471 +0000 UTC m=+40.193747019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/83ceb079-6106-4ef2-a063-3506b4eb60a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vzv57" (UID: "83ceb079-6106-4ef2-a063-3506b4eb60a2") : configmap "kube-rbac-proxy" not found Aug 04 18:11:22.442676 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:22.442595 2526 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 18:11:22.442676 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:22.442630 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls podName:83ceb079-6106-4ef2-a063-3506b4eb60a2 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:22.942620027 +0000 UTC m=+40.193902554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vzv57" (UID: "83ceb079-6106-4ef2-a063-3506b4eb60a2") : secret "insights-runtime-extractor-tls" not found Aug 04 18:11:22.442829 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:22.442677 2526 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 18:11:22.442829 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.442692 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83ceb079-6106-4ef2-a063-3506b4eb60a2-crio-socket\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.442829 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:22.442711 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls podName:7439e0a9-d4de-4b6a-ae78-48a889eff5a1 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:24.442701203 +0000 UTC m=+41.693983730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls") pod "dns-default-kdk4v" (UID: "7439e0a9-d4de-4b6a-ae78-48a889eff5a1") : secret "dns-default-metrics-tls" not found Aug 04 18:11:22.442829 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.442728 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83ceb079-6106-4ef2-a063-3506b4eb60a2-data-volume\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.470424 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.470401 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfkmd\" (UniqueName: \"kubernetes.io/projected/83ceb079-6106-4ef2-a063-3506b4eb60a2-kube-api-access-mfkmd\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.551167 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.551141 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:22.690554 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.690526 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:22.768314 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.768228 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:22.867595 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.867562 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:22.946654 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.946604 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83ceb079-6106-4ef2-a063-3506b4eb60a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.946807 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:22.946662 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:22.946807 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:22.946737 2526 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 18:11:22.946807 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:22.946780 2526 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 18:11:22.946938 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:22.946815 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83ceb079-6106-4ef2-a063-3506b4eb60a2-kube-rbac-proxy-cm podName:83ceb079-6106-4ef2-a063-3506b4eb60a2 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:23.946795162 +0000 UTC m=+41.198077688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/83ceb079-6106-4ef2-a063-3506b4eb60a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vzv57" (UID: "83ceb079-6106-4ef2-a063-3506b4eb60a2") : configmap "kube-rbac-proxy" not found Aug 04 18:11:22.946938 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:22.946834 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls podName:83ceb079-6106-4ef2-a063-3506b4eb60a2 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:23.946824588 +0000 UTC m=+41.198107117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vzv57" (UID: "83ceb079-6106-4ef2-a063-3506b4eb60a2") : secret "insights-runtime-extractor-tls" not found Aug 04 18:11:23.218937 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:23.218808 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:23.316856 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:23.316821 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:23.525786 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:23.525703 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:23.597895 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:23.597865 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:23.808410 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:23.808335 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:23.951585 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:23.951557 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:23.954531 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:23.954512 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83ceb079-6106-4ef2-a063-3506b4eb60a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:23.954636 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:23.954548 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:23.954636 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:23.954628 2526 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 18:11:23.954727 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:23.954695 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83ceb079-6106-4ef2-a063-3506b4eb60a2-kube-rbac-proxy-cm podName:83ceb079-6106-4ef2-a063-3506b4eb60a2 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:25.954676745 +0000 UTC m=+43.205959282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/83ceb079-6106-4ef2-a063-3506b4eb60a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vzv57" (UID: "83ceb079-6106-4ef2-a063-3506b4eb60a2") : configmap "kube-rbac-proxy" not found Aug 04 18:11:23.954783 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:23.954730 2526 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 18:11:23.954783 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:23.954775 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls podName:83ceb079-6106-4ef2-a063-3506b4eb60a2 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:25.954763006 +0000 UTC m=+43.206045531 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vzv57" (UID: "83ceb079-6106-4ef2-a063-3506b4eb60a2") : secret "insights-runtime-extractor-tls" not found Aug 04 18:11:24.095072 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:24.095052 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rvt4h_fb9a403b-8397-4699-bde5-4a19d861d697/dns-node-resolver/0.log" Aug 04 18:11:24.373579 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:24.373494 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xr2hn" event={"ID":"c9e9f16d-731b-4afb-95be-304a641d3e4d","Type":"ContainerStarted","Data":"d2c74162fa0a48e4eea8a5566e9befe8f4d6cc7fde58b50cd4b7e4fe0e32bb78"} Aug 04 18:11:24.391599 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:24.391553 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xr2hn" podStartSLOduration=1.653743127 podStartE2EDuration="4.3915407s" totalImagesPullingTime="2.737797573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:24.390700478 +0000 UTC m=+41.641983024" watchObservedRunningTime="2026-08-04 18:11:24.3915407 +0000 UTC m=+41.642823225" Aug 04 18:11:24.459581 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:24.459551 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:24.459706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:24.459619 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:24.459706 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:24.459696 2526 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 18:11:24.459800 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:24.459754 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls podName:7439e0a9-d4de-4b6a-ae78-48a889eff5a1 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:28.459739505 +0000 UTC m=+45.711022029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls") pod "dns-default-kdk4v" (UID: "7439e0a9-d4de-4b6a-ae78-48a889eff5a1") : secret "dns-default-metrics-tls" not found Aug 04 18:11:24.459800 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:24.459756 2526 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 18:11:24.459800 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:24.459773 2526 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4c467488-kwzr5: secret "image-registry-tls" not found Aug 04 18:11:24.459891 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:24.459811 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls podName:e375a34f-0263-4345-9317-067d37d3a95c nodeName:}" failed. No retries permitted until 2026-08-04 18:11:28.459798953 +0000 UTC m=+45.711081478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls") pod "image-registry-7f4c467488-kwzr5" (UID: "e375a34f-0263-4345-9317-067d37d3a95c") : secret "image-registry-tls" not found Aug 04 18:11:25.095085 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.095056 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p9xc7_884f5581-c681-4785-a065-fa174e052347/node-ca/0.log" Aug 04 18:11:25.095434 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.095319 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-wdgmp"] Aug 04 18:11:25.110161 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.110136 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-wdgmp" Aug 04 18:11:25.110161 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.110149 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-wdgmp"] Aug 04 18:11:25.112524 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.112505 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Aug 04 18:11:25.112677 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.112530 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Aug 04 18:11:25.113714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.113672 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Aug 04 18:11:25.113714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.113693 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-sjhkz\"" Aug 04 18:11:25.113714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.113698 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Aug 04 18:11:25.135418 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.135264 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:25.212682 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.212657 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:25.265958 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.265938 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c1f4ffd4-3a88-416a-a2b7-fc887bf72230-signing-cabundle\") pod \"service-ca-98c88fbb5-wdgmp\" (UID: \"c1f4ffd4-3a88-416a-a2b7-fc887bf72230\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdgmp" Aug 04 18:11:25.266044 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.265999 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cm9z\" (UniqueName: \"kubernetes.io/projected/c1f4ffd4-3a88-416a-a2b7-fc887bf72230-kube-api-access-4cm9z\") pod \"service-ca-98c88fbb5-wdgmp\" (UID: \"c1f4ffd4-3a88-416a-a2b7-fc887bf72230\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdgmp" Aug 04 18:11:25.266091 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.266058 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c1f4ffd4-3a88-416a-a2b7-fc887bf72230-signing-key\") pod \"service-ca-98c88fbb5-wdgmp\" (UID: \"c1f4ffd4-3a88-416a-a2b7-fc887bf72230\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdgmp" Aug 04 18:11:25.367400 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.367275 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c1f4ffd4-3a88-416a-a2b7-fc887bf72230-signing-cabundle\") pod \"service-ca-98c88fbb5-wdgmp\" (UID: \"c1f4ffd4-3a88-416a-a2b7-fc887bf72230\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdgmp" Aug 04 18:11:25.367400 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.367391 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cm9z\" (UniqueName: \"kubernetes.io/projected/c1f4ffd4-3a88-416a-a2b7-fc887bf72230-kube-api-access-4cm9z\") pod \"service-ca-98c88fbb5-wdgmp\" (UID: \"c1f4ffd4-3a88-416a-a2b7-fc887bf72230\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdgmp" Aug 04 18:11:25.367546 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.367453 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c1f4ffd4-3a88-416a-a2b7-fc887bf72230-signing-key\") pod \"service-ca-98c88fbb5-wdgmp\" (UID: \"c1f4ffd4-3a88-416a-a2b7-fc887bf72230\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdgmp" Aug 04 18:11:25.367864 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.367842 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c1f4ffd4-3a88-416a-a2b7-fc887bf72230-signing-cabundle\") pod \"service-ca-98c88fbb5-wdgmp\" (UID: \"c1f4ffd4-3a88-416a-a2b7-fc887bf72230\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdgmp" Aug 04 18:11:25.370654 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.370637 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c1f4ffd4-3a88-416a-a2b7-fc887bf72230-signing-key\") pod \"service-ca-98c88fbb5-wdgmp\" (UID: \"c1f4ffd4-3a88-416a-a2b7-fc887bf72230\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdgmp" Aug 04 18:11:25.376646 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.376626 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cm9z\" (UniqueName: \"kubernetes.io/projected/c1f4ffd4-3a88-416a-a2b7-fc887bf72230-kube-api-access-4cm9z\") pod \"service-ca-98c88fbb5-wdgmp\" (UID: \"c1f4ffd4-3a88-416a-a2b7-fc887bf72230\") " pod="openshift-service-ca/service-ca-98c88fbb5-wdgmp" Aug 04 18:11:25.419461 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.419440 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-wdgmp" Aug 04 18:11:25.554768 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.554740 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-wdgmp"] Aug 04 18:11:25.558837 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:11:25.558815 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f4ffd4_3a88_416a_a2b7_fc887bf72230.slice/crio-85649eb1b5d3fcbcd2e3e7f8cd2cb656c8a314acb1e7706409b0a63afcd5b683 WatchSource:0}: Error finding container 85649eb1b5d3fcbcd2e3e7f8cd2cb656c8a314acb1e7706409b0a63afcd5b683: Status 404 returned error can't find the container with id 85649eb1b5d3fcbcd2e3e7f8cd2cb656c8a314acb1e7706409b0a63afcd5b683 Aug 04 18:11:25.911429 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.911386 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:25.973470 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.973443 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83ceb079-6106-4ef2-a063-3506b4eb60a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:25.973581 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.973473 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:25.973665 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:25.973652 2526 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 18:11:25.973711 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:25.973702 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls podName:83ceb079-6106-4ef2-a063-3506b4eb60a2 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:29.973689815 +0000 UTC m=+47.224972339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vzv57" (UID: "83ceb079-6106-4ef2-a063-3506b4eb60a2") : secret "insights-runtime-extractor-tls" not found Aug 04 18:11:25.973948 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:25.973931 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83ceb079-6106-4ef2-a063-3506b4eb60a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:26.019757 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:26.019732 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:26.094984 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:26.094948 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xr2hn_c9e9f16d-731b-4afb-95be-304a641d3e4d/serve-healthcheck-canary/0.log" Aug 04 18:11:26.121062 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:26.121029 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:26.380031 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:26.379994 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-wdgmp" event={"ID":"c1f4ffd4-3a88-416a-a2b7-fc887bf72230","Type":"ContainerStarted","Data":"85649eb1b5d3fcbcd2e3e7f8cd2cb656c8a314acb1e7706409b0a63afcd5b683"} Aug 04 18:11:28.387621 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:28.387266 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-wdgmp" event={"ID":"c1f4ffd4-3a88-416a-a2b7-fc887bf72230","Type":"ContainerStarted","Data":"cedfaea260f1b4aa81315b3c7982fcf8cdfacb29a4b26dda85d26f8df7da318e"} Aug 04 18:11:28.405271 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:28.405228 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-wdgmp" podStartSLOduration=0.834852552 podStartE2EDuration="3.405214363s" totalImagesPullingTime="2.570361811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:28.404519795 +0000 UTC m=+45.655802342" watchObservedRunningTime="2026-08-04 18:11:28.405214363 +0000 UTC m=+45.656496910" Aug 04 18:11:28.490007 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:28.489978 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:28.490149 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:28.490067 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:28.490149 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:28.490139 2526 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 18:11:28.490253 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:28.490149 2526 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 18:11:28.490253 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:28.490163 2526 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4c467488-kwzr5: secret "image-registry-tls" not found Aug 04 18:11:28.490253 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:28.490202 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls podName:7439e0a9-d4de-4b6a-ae78-48a889eff5a1 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:36.490184781 +0000 UTC m=+53.741467319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls") pod "dns-default-kdk4v" (UID: "7439e0a9-d4de-4b6a-ae78-48a889eff5a1") : secret "dns-default-metrics-tls" not found Aug 04 18:11:28.490253 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:28.490221 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls podName:e375a34f-0263-4345-9317-067d37d3a95c nodeName:}" failed. No retries permitted until 2026-08-04 18:11:36.490212791 +0000 UTC m=+53.741495316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls") pod "image-registry-7f4c467488-kwzr5" (UID: "e375a34f-0263-4345-9317-067d37d3a95c") : secret "image-registry-tls" not found Aug 04 18:11:29.453146 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:29.453104 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:29.624088 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:29.624049 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:30.004498 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:30.004463 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:30.004690 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:30.004626 2526 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 18:11:30.004690 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:11:30.004688 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls podName:83ceb079-6106-4ef2-a063-3506b4eb60a2 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:38.004669838 +0000 UTC m=+55.255952377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vzv57" (UID: "83ceb079-6106-4ef2-a063-3506b4eb60a2") : secret "insights-runtime-extractor-tls" not found Aug 04 18:11:30.060632 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:30.060601 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:30.262015 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:30.261917 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:30.365753 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:30.365722 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:30.640735 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:30.640663 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:30.742130 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:30.742104 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:31.134485 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:31.134453 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:31.576996 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:31.576951 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:31.776681 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:31.776653 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:31.955502 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:31.955471 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:32.156646 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:32.156613 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:33.066859 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:33.066830 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:33.170524 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:33.170488 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:33.513567 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:33.513518 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:33.739437 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:33.739390 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:33.946465 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:33.946388 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:34.025768 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:34.025738 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:34.311357 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:34.311327 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:34.794903 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:34.794873 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:35.532535 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:35.532497 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:35.744986 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:35.744934 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:35.839607 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:35.839527 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:36.116594 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:36.116487 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:36.549942 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:36.549913 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:36.550381 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:36.550007 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:36.553098 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:36.553079 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls\") pod \"image-registry-7f4c467488-kwzr5\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:36.553174 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:36.553153 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7439e0a9-d4de-4b6a-ae78-48a889eff5a1-metrics-tls\") pod \"dns-default-kdk4v\" (UID: \"7439e0a9-d4de-4b6a-ae78-48a889eff5a1\") " pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:36.555631 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:36.555603 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:36.602667 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:36.602639 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:36.620732 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:36.620709 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:36.638616 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:36.638588 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:36.791674 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:36.791654 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f4c467488-kwzr5"] Aug 04 18:11:36.793578 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:11:36.793555 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode375a34f_0263_4345_9317_067d37d3a95c.slice/crio-7b886ab14e90557df9e7688577be32d2793bf6051b2615cc11ead4804af6ab15 WatchSource:0}: Error finding container 7b886ab14e90557df9e7688577be32d2793bf6051b2615cc11ead4804af6ab15: Status 404 returned error can't find the container with id 7b886ab14e90557df9e7688577be32d2793bf6051b2615cc11ead4804af6ab15 Aug 04 18:11:36.829437 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:36.829162 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kdk4v"] Aug 04 18:11:36.832126 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:11:36.832101 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7439e0a9_d4de_4b6a_ae78_48a889eff5a1.slice/crio-436cb4e03f74460272f774ac9eebf4bcdf5f14243f0fa901aab7e6ed21b655a9 WatchSource:0}: Error finding container 436cb4e03f74460272f774ac9eebf4bcdf5f14243f0fa901aab7e6ed21b655a9: Status 404 returned error can't find the container with id 436cb4e03f74460272f774ac9eebf4bcdf5f14243f0fa901aab7e6ed21b655a9 Aug 04 18:11:36.839268 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:36.839241 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:36.925926 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:36.925882 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:37.412845 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:37.412808 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" event={"ID":"e375a34f-0263-4345-9317-067d37d3a95c","Type":"ContainerStarted","Data":"fa87d3c4d00aa10d24e48116d1842e4ffb8001e5aa7d42f8c28f162e545d6548"} Aug 04 18:11:37.413056 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:37.412857 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" event={"ID":"e375a34f-0263-4345-9317-067d37d3a95c","Type":"ContainerStarted","Data":"7b886ab14e90557df9e7688577be32d2793bf6051b2615cc11ead4804af6ab15"} Aug 04 18:11:37.413056 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:37.412883 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:37.413993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:37.413955 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kdk4v" event={"ID":"7439e0a9-d4de-4b6a-ae78-48a889eff5a1","Type":"ContainerStarted","Data":"436cb4e03f74460272f774ac9eebf4bcdf5f14243f0fa901aab7e6ed21b655a9"} Aug 04 18:11:37.436222 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:37.436177 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" podStartSLOduration=27.436161773 podStartE2EDuration="27.436161773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:37.435373421 +0000 UTC m=+54.686655967" watchObservedRunningTime="2026-08-04 18:11:37.436161773 +0000 UTC m=+54.687444298" Aug 04 18:11:37.518036 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:37.518012 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:37.721247 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:37.721216 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:38.059764 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:38.059672 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:38.061608 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:38.061581 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:38.064186 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:38.064165 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83ceb079-6106-4ef2-a063-3506b4eb60a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vzv57\" (UID: \"83ceb079-6106-4ef2-a063-3506b4eb60a2\") " pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:38.117636 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:38.117491 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vzv57" Aug 04 18:11:38.160450 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:38.160422 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:38.271583 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:38.271553 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vzv57"] Aug 04 18:11:38.274957 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:11:38.274928 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ceb079_6106_4ef2_a063_3506b4eb60a2.slice/crio-80138576763ffb799661d3dfa5f080c867f5042d97153804c0eb4c7ba6cb45f8 WatchSource:0}: Error finding container 80138576763ffb799661d3dfa5f080c867f5042d97153804c0eb4c7ba6cb45f8: Status 404 returned error can't find the container with id 80138576763ffb799661d3dfa5f080c867f5042d97153804c0eb4c7ba6cb45f8 Aug 04 18:11:38.312355 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:38.312297 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:38.418486 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:38.418451 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vzv57" event={"ID":"83ceb079-6106-4ef2-a063-3506b4eb60a2","Type":"ContainerStarted","Data":"b36442639e02edcd50d6bc0966cd21e249b12189a2138fba45248ca3ab74cb9e"} Aug 04 18:11:38.418616 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:38.418519 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vzv57" event={"ID":"83ceb079-6106-4ef2-a063-3506b4eb60a2","Type":"ContainerStarted","Data":"80138576763ffb799661d3dfa5f080c867f5042d97153804c0eb4c7ba6cb45f8"} Aug 04 18:11:38.731691 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:38.731655 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:39.335860 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:39.335826 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:39.422727 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:39.422700 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kdk4v" event={"ID":"7439e0a9-d4de-4b6a-ae78-48a889eff5a1","Type":"ContainerStarted","Data":"750b6f5021c96dc5dee6327969c833d44072e00924d56ace0d52a2b30d1ca403"} Aug 04 18:11:39.422849 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:39.422749 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kdk4v" event={"ID":"7439e0a9-d4de-4b6a-ae78-48a889eff5a1","Type":"ContainerStarted","Data":"e0df3c708001645311deda3d1c6ccce1841a44a273f932e186c196fbd82fe895"} Aug 04 18:11:39.422849 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:39.422804 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:39.424328 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:39.424309 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vzv57" event={"ID":"83ceb079-6106-4ef2-a063-3506b4eb60a2","Type":"ContainerStarted","Data":"2e6c3acfa84bf3170267356c7210dafa03725783d8834324014f47b5c0eaa8db"} Aug 04 18:11:39.441649 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:39.441602 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kdk4v" podStartSLOduration=17.270013477 podStartE2EDuration="19.441585406s" totalImagesPullingTime="2.171571929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:39.44104694 +0000 UTC m=+56.692329486" watchObservedRunningTime="2026-08-04 18:11:39.441585406 +0000 UTC m=+56.692867955" Aug 04 18:11:39.710892 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:39.710858 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:39.934397 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:39.934343 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:40.129538 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:40.129456 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:41.433902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:41.433814 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vzv57" event={"ID":"83ceb079-6106-4ef2-a063-3506b4eb60a2","Type":"ContainerStarted","Data":"b3aa5572a7b3dd490765f1bec4d93003cf7ea6e502ac22a4107a8c675cee9fcf"} Aug 04 18:11:41.456129 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:41.456085 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vzv57" podStartSLOduration=16.844457774 podStartE2EDuration="19.45607246s" totalImagesPullingTime="2.611614686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:11:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:41.455540259 +0000 UTC m=+58.706822818" watchObservedRunningTime="2026-08-04 18:11:41.45607246 +0000 UTC m=+58.707355007" Aug 04 18:11:44.782776 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:44.782739 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:44.880323 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:44.880292 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:44.982284 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:44.982252 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:45.082052 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:45.081959 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:45.348140 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:45.348061 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ns2w" Aug 04 18:11:45.432405 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:45.432377 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:45.630061 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:45.629985 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:45.696785 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:45.696755 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:45.864585 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:45.864555 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:45.955471 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:45.955431 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:46.261331 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:46.261247 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:46.455745 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:46.455703 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:46.556416 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:46.556330 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:47.481903 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:47.481866 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:47.620072 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:47.620041 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:47.761115 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:47.761036 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:47.929765 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:47.929734 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:48.358053 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.358025 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:48.486120 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.486087 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x"] Aug 04 18:11:48.490046 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.490022 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" Aug 04 18:11:48.491487 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.491464 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:48.495827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.495807 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 04 18:11:48.496214 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.496190 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 04 18:11:48.496394 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.496355 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Aug 04 18:11:48.496394 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.496382 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f4c467488-kwzr5"] Aug 04 18:11:48.497249 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.497229 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 04 18:11:48.509386 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.509365 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x"] Aug 04 18:11:48.536927 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.536901 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f"] Aug 04 18:11:48.538742 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.538719 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.539653 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.539632 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7qt7\" (UniqueName: \"kubernetes.io/projected/2f142fd3-9be5-443b-b859-1ebec03ea6e1-kube-api-access-q7qt7\") pod \"klusterlet-addon-workmgr-6fcb7c7897-6mf2x\" (UID: \"2f142fd3-9be5-443b-b859-1ebec03ea6e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" Aug 04 18:11:48.539736 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.539667 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2f142fd3-9be5-443b-b859-1ebec03ea6e1-klusterlet-config\") pod \"klusterlet-addon-workmgr-6fcb7c7897-6mf2x\" (UID: \"2f142fd3-9be5-443b-b859-1ebec03ea6e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" Aug 04 18:11:48.539736 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.539705 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f142fd3-9be5-443b-b859-1ebec03ea6e1-tmp\") pod \"klusterlet-addon-workmgr-6fcb7c7897-6mf2x\" (UID: \"2f142fd3-9be5-443b-b859-1ebec03ea6e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" Aug 04 18:11:48.542870 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.542856 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Aug 04 18:11:48.543096 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.543082 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Aug 04 18:11:48.543215 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.543192 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Aug 04 18:11:48.543551 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.543534 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Aug 04 18:11:48.559318 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.559299 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f"] Aug 04 18:11:48.640193 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.640128 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d9293c96-f16c-45c0-8b15-accbe49c5281-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.640193 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.640154 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9293c96-f16c-45c0-8b15-accbe49c5281-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.640193 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.640179 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d9293c96-f16c-45c0-8b15-accbe49c5281-ca\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.640340 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.640212 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d9293c96-f16c-45c0-8b15-accbe49c5281-hub\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.640340 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.640266 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2f142fd3-9be5-443b-b859-1ebec03ea6e1-klusterlet-config\") pod \"klusterlet-addon-workmgr-6fcb7c7897-6mf2x\" (UID: \"2f142fd3-9be5-443b-b859-1ebec03ea6e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" Aug 04 18:11:48.640340 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.640314 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg5h4\" (UniqueName: \"kubernetes.io/projected/d9293c96-f16c-45c0-8b15-accbe49c5281-kube-api-access-rg5h4\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.640452 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.640347 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7qt7\" (UniqueName: \"kubernetes.io/projected/2f142fd3-9be5-443b-b859-1ebec03ea6e1-kube-api-access-q7qt7\") pod \"klusterlet-addon-workmgr-6fcb7c7897-6mf2x\" (UID: \"2f142fd3-9be5-443b-b859-1ebec03ea6e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" Aug 04 18:11:48.640452 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.640367 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f142fd3-9be5-443b-b859-1ebec03ea6e1-tmp\") pod \"klusterlet-addon-workmgr-6fcb7c7897-6mf2x\" (UID: \"2f142fd3-9be5-443b-b859-1ebec03ea6e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" Aug 04 18:11:48.640452 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.640440 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9293c96-f16c-45c0-8b15-accbe49c5281-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.640716 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.640698 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f142fd3-9be5-443b-b859-1ebec03ea6e1-tmp\") pod \"klusterlet-addon-workmgr-6fcb7c7897-6mf2x\" (UID: \"2f142fd3-9be5-443b-b859-1ebec03ea6e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" Aug 04 18:11:48.642593 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.642573 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2f142fd3-9be5-443b-b859-1ebec03ea6e1-klusterlet-config\") pod \"klusterlet-addon-workmgr-6fcb7c7897-6mf2x\" (UID: \"2f142fd3-9be5-443b-b859-1ebec03ea6e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" Aug 04 18:11:48.654798 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.654774 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:48.656275 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.656256 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7qt7\" (UniqueName: \"kubernetes.io/projected/2f142fd3-9be5-443b-b859-1ebec03ea6e1-kube-api-access-q7qt7\") pod \"klusterlet-addon-workmgr-6fcb7c7897-6mf2x\" (UID: \"2f142fd3-9be5-443b-b859-1ebec03ea6e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" Aug 04 18:11:48.741318 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.741294 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9293c96-f16c-45c0-8b15-accbe49c5281-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.741415 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.741324 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d9293c96-f16c-45c0-8b15-accbe49c5281-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.741415 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.741343 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9293c96-f16c-45c0-8b15-accbe49c5281-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.741415 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.741358 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d9293c96-f16c-45c0-8b15-accbe49c5281-ca\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.741415 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.741374 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d9293c96-f16c-45c0-8b15-accbe49c5281-hub\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.741415 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.741403 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg5h4\" (UniqueName: \"kubernetes.io/projected/d9293c96-f16c-45c0-8b15-accbe49c5281-kube-api-access-rg5h4\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.742110 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.742087 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9293c96-f16c-45c0-8b15-accbe49c5281-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.743503 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.743474 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d9293c96-f16c-45c0-8b15-accbe49c5281-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.743673 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.743654 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d9293c96-f16c-45c0-8b15-accbe49c5281-ca\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.743785 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.743769 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d9293c96-f16c-45c0-8b15-accbe49c5281-hub\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.743822 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.743802 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9293c96-f16c-45c0-8b15-accbe49c5281-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.754892 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.754875 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg5h4\" (UniqueName: \"kubernetes.io/projected/d9293c96-f16c-45c0-8b15-accbe49c5281-kube-api-access-rg5h4\") pod \"cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f\" (UID: \"d9293c96-f16c-45c0-8b15-accbe49c5281\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.757768 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.757749 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:48.800062 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.800046 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" Aug 04 18:11:48.842526 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.842496 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs\") pod \"network-metrics-daemon-ntsjj\" (UID: \"11d07db5-806d-44e7-af32-be9ae461cbae\") " pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:48.845084 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.845066 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 18:11:48.856264 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.856130 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:48.856264 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.856251 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11d07db5-806d-44e7-af32-be9ae461cbae-metrics-certs\") pod \"network-metrics-daemon-ntsjj\" (UID: \"11d07db5-806d-44e7-af32-be9ae461cbae\") " pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:48.860267 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.860226 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" Aug 04 18:11:48.899097 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.899015 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wz6f8\"" Aug 04 18:11:48.907270 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.907244 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntsjj" Aug 04 18:11:48.927720 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.927695 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x"] Aug 04 18:11:48.930545 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:11:48.930519 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f142fd3_9be5_443b_b859_1ebec03ea6e1.slice/crio-ecf2c6c1bf03f0033b209fd9c03df672bc727f2f7939bf51905e43ff0e9e457b WatchSource:0}: Error finding container ecf2c6c1bf03f0033b209fd9c03df672bc727f2f7939bf51905e43ff0e9e457b: Status 404 returned error can't find the container with id ecf2c6c1bf03f0033b209fd9c03df672bc727f2f7939bf51905e43ff0e9e457b Aug 04 18:11:48.943581 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.943556 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:48.945907 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.945884 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 18:11:48.956909 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.956854 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94ea3648-f325-45aa-b53b-f5c59bd1bd5a-original-pull-secret\") pod \"global-pull-secret-syncer-r59lx\" (UID: \"94ea3648-f325-45aa-b53b-f5c59bd1bd5a\") " pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:48.957824 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.957791 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:48.997631 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:48.997606 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f"] Aug 04 18:11:49.003583 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:11:49.003555 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9293c96_f16c_45c0_8b15_accbe49c5281.slice/crio-653d70e45900be13e614f2c620db2812f9c227697e3c48bceb4eb9b9fbc365e3 WatchSource:0}: Error finding container 653d70e45900be13e614f2c620db2812f9c227697e3c48bceb4eb9b9fbc365e3: Status 404 returned error can't find the container with id 653d70e45900be13e614f2c620db2812f9c227697e3c48bceb4eb9b9fbc365e3 Aug 04 18:11:49.035370 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.035347 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ntsjj"] Aug 04 18:11:49.038192 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:11:49.038170 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d07db5_806d_44e7_af32_be9ae461cbae.slice/crio-230c967b715bc7361954bdd9ef6a560ada09848c2dbcc8747c842d4b1701ff8d WatchSource:0}: Error finding container 230c967b715bc7361954bdd9ef6a560ada09848c2dbcc8747c842d4b1701ff8d: Status 404 returned error can't find the container with id 230c967b715bc7361954bdd9ef6a560ada09848c2dbcc8747c842d4b1701ff8d Aug 04 18:11:49.039430 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.039405 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:49.044739 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.044720 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk88z\" (UniqueName: \"kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z\") pod \"network-check-target-qmq4s\" (UID: \"0119f70f-109c-4ae9-b44e-2c3f69e61821\") " pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:49.047131 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.047115 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 18:11:49.057526 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.057510 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 18:11:49.067699 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.067677 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk88z\" (UniqueName: \"kubernetes.io/projected/0119f70f-109c-4ae9-b44e-2c3f69e61821-kube-api-access-vk88z\") pod \"network-check-target-qmq4s\" (UID: \"0119f70f-109c-4ae9-b44e-2c3f69e61821\") " pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:49.190541 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.190486 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r59lx" Aug 04 18:11:49.205409 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.205390 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kp5sr\"" Aug 04 18:11:49.213388 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.213371 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:49.325621 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.325590 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r59lx"] Aug 04 18:11:49.328609 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:11:49.328581 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ea3648_f325_45aa_b53b_f5c59bd1bd5a.slice/crio-f5fea8923f82c68817ec4078ae5c9ee57ceadc917d0e8e9c925f4948ea145487 WatchSource:0}: Error finding container f5fea8923f82c68817ec4078ae5c9ee57ceadc917d0e8e9c925f4948ea145487: Status 404 returned error can't find the container with id f5fea8923f82c68817ec4078ae5c9ee57ceadc917d0e8e9c925f4948ea145487 Aug 04 18:11:49.352520 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.352494 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qmq4s"] Aug 04 18:11:49.355418 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:11:49.355396 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0119f70f_109c_4ae9_b44e_2c3f69e61821.slice/crio-baed9847c6ddc902ad5dd84049cc4bd12cbcf28c100b73601a8f943b09890437 WatchSource:0}: Error finding container baed9847c6ddc902ad5dd84049cc4bd12cbcf28c100b73601a8f943b09890437: Status 404 returned error can't find the container with id baed9847c6ddc902ad5dd84049cc4bd12cbcf28c100b73601a8f943b09890437 Aug 04 18:11:49.431233 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.431211 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kdk4v" Aug 04 18:11:49.458206 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.458185 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ntsjj" event={"ID":"11d07db5-806d-44e7-af32-be9ae461cbae","Type":"ContainerStarted","Data":"230c967b715bc7361954bdd9ef6a560ada09848c2dbcc8747c842d4b1701ff8d"} Aug 04 18:11:49.459214 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.459191 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" event={"ID":"d9293c96-f16c-45c0-8b15-accbe49c5281","Type":"ContainerStarted","Data":"653d70e45900be13e614f2c620db2812f9c227697e3c48bceb4eb9b9fbc365e3"} Aug 04 18:11:49.460228 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.460210 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r59lx" event={"ID":"94ea3648-f325-45aa-b53b-f5c59bd1bd5a","Type":"ContainerStarted","Data":"f5fea8923f82c68817ec4078ae5c9ee57ceadc917d0e8e9c925f4948ea145487"} Aug 04 18:11:49.461275 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.461248 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" event={"ID":"2f142fd3-9be5-443b-b859-1ebec03ea6e1","Type":"ContainerStarted","Data":"ecf2c6c1bf03f0033b209fd9c03df672bc727f2f7939bf51905e43ff0e9e457b"} Aug 04 18:11:49.462178 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:49.462160 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qmq4s" event={"ID":"0119f70f-109c-4ae9-b44e-2c3f69e61821","Type":"ContainerStarted","Data":"baed9847c6ddc902ad5dd84049cc4bd12cbcf28c100b73601a8f943b09890437"} Aug 04 18:11:50.470245 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:50.470090 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ntsjj" event={"ID":"11d07db5-806d-44e7-af32-be9ae461cbae","Type":"ContainerStarted","Data":"0e5efdf821e5219b03ebbebd986a2825c3749ec83511f0ae063f2bf0dde649a1"} Aug 04 18:11:51.478589 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:51.478549 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ntsjj" event={"ID":"11d07db5-806d-44e7-af32-be9ae461cbae","Type":"ContainerStarted","Data":"775e963fba1be8fa9557113f638be486cc536eeff9a211a4800b50c88a3f696c"} Aug 04 18:11:51.498398 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:51.497824 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ntsjj" podStartSLOduration=67.311556114 podStartE2EDuration="1m8.497803746s" totalImagesPullingTime="1.186247632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:51.497225236 +0000 UTC m=+68.748507783" watchObservedRunningTime="2026-08-04 18:11:51.497803746 +0000 UTC m=+68.749086313" Aug 04 18:11:51.596848 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:51.596816 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:51.700646 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:51.700606 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:51.868311 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:51.868054 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:51.949993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:51.949189 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:52.153873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:52.153768 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:52.298348 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:52.298287 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:52.361894 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:52.361851 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:54.646004 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:54.645955 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:54.743885 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:54.743841 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:55.277237 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:55.277197 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:56.501806 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:56.501520 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" event={"ID":"d9293c96-f16c-45c0-8b15-accbe49c5281","Type":"ContainerStarted","Data":"a52965a51aa143fc92c92dcc96bb98f253a2e5f2ae516a5e1d3d50ae26a94419"} Aug 04 18:11:56.501806 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:56.501569 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" event={"ID":"d9293c96-f16c-45c0-8b15-accbe49c5281","Type":"ContainerStarted","Data":"34145db475094299fd36495ae40ed5d584d816087941ae62d908d2cafb63cf1f"} Aug 04 18:11:56.501806 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:56.501589 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" event={"ID":"d9293c96-f16c-45c0-8b15-accbe49c5281","Type":"ContainerStarted","Data":"87b0005166f0c2ae7a3c65080f710ac099984cb48d2d35a4e6dc8f86916222f4"} Aug 04 18:11:56.503750 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:56.503717 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r59lx" event={"ID":"94ea3648-f325-45aa-b53b-f5c59bd1bd5a","Type":"ContainerStarted","Data":"ad84fff5170e5d8df386d18f267b63259f9a9e8c2773d039609d757bfc0d623d"} Aug 04 18:11:56.506139 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:56.506109 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qmq4s" event={"ID":"0119f70f-109c-4ae9-b44e-2c3f69e61821","Type":"ContainerStarted","Data":"51a0aba2fda9f5a97ba59cd76ace0bb2a17ba0911208351e58b6dc8130362a29"} Aug 04 18:11:56.506268 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:56.506255 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:11:56.524144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:56.524086 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f7cb4f4dc-msh8f" podStartSLOduration=1.6771107170000001 podStartE2EDuration="8.524047578s" totalImagesPullingTime="6.846936861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:56.522911734 +0000 UTC m=+73.774194285" watchObservedRunningTime="2026-08-04 18:11:56.524047578 +0000 UTC m=+73.775330126" Aug 04 18:11:56.545988 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:56.545131 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qmq4s" podStartSLOduration=67.067379796 podStartE2EDuration="1m13.545110934s" totalImagesPullingTime="6.477731138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:56.544312203 +0000 UTC m=+73.795594750" watchObservedRunningTime="2026-08-04 18:11:56.545110934 +0000 UTC m=+73.796393486" Aug 04 18:11:56.901155 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:56.901074 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:57.248678 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.248607 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:57.370997 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.370892 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-r59lx" podStartSLOduration=66.848286024 podStartE2EDuration="1m13.370868811s" totalImagesPullingTime="6.522582787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:56.562757854 +0000 UTC m=+73.814040401" watchObservedRunningTime="2026-08-04 18:11:57.370868811 +0000 UTC m=+74.622151340" Aug 04 18:11:57.373251 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.371994 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kpskw"] Aug 04 18:11:57.382615 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.382584 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.387076 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.386811 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 18:11:57.389244 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.388712 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wk8r4\"" Aug 04 18:11:57.396874 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.396446 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 18:11:57.396874 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.396664 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 18:11:57.397631 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.397358 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 18:11:57.397631 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.397562 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 18:11:57.405654 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.405525 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 18:11:57.440688 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.440608 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:57.510557 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.510474 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-root\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.510557 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.510518 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdn2k\" (UniqueName: \"kubernetes.io/projected/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-kube-api-access-jdn2k\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.511089 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.510561 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-node-exporter-tls\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.511089 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.510587 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-node-exporter-wtmp\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.511089 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.510682 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.511089 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.510707 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-metrics-client-ca\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.511089 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.510731 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.511089 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.510768 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-node-exporter-textfile\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.511089 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.510836 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-sys\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.561986 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.558532 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:57.612480 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.611277 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-root\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.612480 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.611335 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdn2k\" (UniqueName: \"kubernetes.io/projected/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-kube-api-access-jdn2k\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.612480 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.611400 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-node-exporter-tls\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.612480 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.611429 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-node-exporter-wtmp\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.612480 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.611453 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.612480 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.611477 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-metrics-client-ca\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.612480 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.611497 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.612480 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.611549 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-node-exporter-textfile\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.612480 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.611591 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-sys\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.612480 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.611692 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-sys\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.612480 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.612169 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-node-exporter-wtmp\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.614718 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.612734 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-root\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.614718 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.613070 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-metrics-client-ca\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.614718 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.613421 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-node-exporter-textfile\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.614718 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.613501 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.618724 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.618692 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-node-exporter-tls\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.623735 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.623692 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdn2k\" (UniqueName: \"kubernetes.io/projected/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-kube-api-access-jdn2k\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.627367 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.627338 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/020e5d5d-aaab-439c-8927-65a1aeb4c9a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kpskw\" (UID: \"020e5d5d-aaab-439c-8927-65a1aeb4c9a1\") " pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.652489 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.652460 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:57.697350 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.697328 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kpskw" Aug 04 18:11:57.709199 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:11:57.709170 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020e5d5d_aaab_439c_8927_65a1aeb4c9a1.slice/crio-7a36a1594c117aaea4300def3c68bba71dfbc77ac1e0488cd17d5ec934c1d59e WatchSource:0}: Error finding container 7a36a1594c117aaea4300def3c68bba71dfbc77ac1e0488cd17d5ec934c1d59e: Status 404 returned error can't find the container with id 7a36a1594c117aaea4300def3c68bba71dfbc77ac1e0488cd17d5ec934c1d59e Aug 04 18:11:57.753087 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.752904 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:57.853607 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.853541 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:57.954542 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:57.954502 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:58.068704 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:58.068671 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:58.250600 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:58.250562 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:58.371788 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:58.371748 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:58.502726 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:58.502654 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:11:58.517393 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:58.517365 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kpskw" event={"ID":"020e5d5d-aaab-439c-8927-65a1aeb4c9a1","Type":"ContainerStarted","Data":"7a36a1594c117aaea4300def3c68bba71dfbc77ac1e0488cd17d5ec934c1d59e"} Aug 04 18:11:58.637711 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:58.637674 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:58.733766 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:58.733723 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:11:59.522942 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:59.522906 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" event={"ID":"2f142fd3-9be5-443b-b859-1ebec03ea6e1","Type":"ContainerStarted","Data":"bda416e9360557f1ba4310f36d3d7a029b7ad91b9ea01d0a94a191ae8ac4709a"} Aug 04 18:11:59.523345 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:59.523068 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" Aug 04 18:11:59.524793 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:59.524773 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" Aug 04 18:11:59.552201 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:11:59.550388 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fcb7c7897-6mf2x" podStartSLOduration=1.350783937 podStartE2EDuration="11.550370124s" totalImagesPullingTime="10.199586187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:59.549945425 +0000 UTC m=+76.801227974" watchObservedRunningTime="2026-08-04 18:11:59.550370124 +0000 UTC m=+76.801652674" Aug 04 18:12:00.125518 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:00.125484 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:00.261048 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:00.260995 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:00.365312 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:00.365281 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:00.512387 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:00.512308 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:00.527438 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:00.527411 2526 generic.go:356] "Generic (PLEG): container finished" podID="020e5d5d-aaab-439c-8927-65a1aeb4c9a1" containerID="f1fdc07c6da26cfb213579b544c95cf0dcc3a88c01652a2bca46b79a7d3799c1" exitCode=0 Aug 04 18:12:00.527778 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:00.527496 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kpskw" event={"ID":"020e5d5d-aaab-439c-8927-65a1aeb4c9a1","Type":"ContainerDied","Data":"f1fdc07c6da26cfb213579b544c95cf0dcc3a88c01652a2bca46b79a7d3799c1"} Aug 04 18:12:01.533214 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:01.533177 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kpskw" event={"ID":"020e5d5d-aaab-439c-8927-65a1aeb4c9a1","Type":"ContainerStarted","Data":"d93f83a4322ef57ce5de90bf5e7cd1170831eeca178fdbb9decd881dd60e7fa4"} Aug 04 18:12:01.533593 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:01.533235 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kpskw" event={"ID":"020e5d5d-aaab-439c-8927-65a1aeb4c9a1","Type":"ContainerStarted","Data":"a943e3d2666bc4841e9f5ec404005fb4337be961e6771ae3c83d3dfef674f60b"} Aug 04 18:12:01.581618 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:01.581576 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kpskw" podStartSLOduration=2.304220467 podStartE2EDuration="4.581560175s" totalImagesPullingTime="2.214324053s" totalInitContainerRuntime="63.015655ms" isStatefulPod=true podCreationTimestamp="2026-08-04 18:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:01.579765331 +0000 UTC m=+78.831047890" watchObservedRunningTime="2026-08-04 18:12:01.581560175 +0000 UTC m=+78.832842722" Aug 04 18:12:01.989566 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:01.989530 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:02.101710 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.101649 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:02.583628 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.583599 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-75df96dc46-88sdg"] Aug 04 18:12:02.586952 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.586936 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.589625 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.589603 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Aug 04 18:12:02.590358 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.590344 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Aug 04 18:12:02.590789 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.590766 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Aug 04 18:12:02.590887 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.590856 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-cx9l9\"" Aug 04 18:12:02.590950 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.590941 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Aug 04 18:12:02.591034 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.591009 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Aug 04 18:12:02.595264 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.595246 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Aug 04 18:12:02.613663 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.613639 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75df96dc46-88sdg"] Aug 04 18:12:02.757626 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.757590 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-serving-certs-ca-bundle\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.757788 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.757642 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-metrics-client-ca\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.757788 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.757697 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-federate-client-tls\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.757788 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.757735 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.757900 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.757800 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-secret-telemeter-client\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.757900 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.757825 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ng8\" (UniqueName: \"kubernetes.io/projected/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-kube-api-access-p8ng8\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.757900 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.757855 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.758038 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.757885 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-telemeter-client-tls\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.858989 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.858881 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-metrics-client-ca\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.858989 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.858935 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-federate-client-tls\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.858989 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.858955 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.859192 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.859028 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-secret-telemeter-client\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.859192 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.859054 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ng8\" (UniqueName: \"kubernetes.io/projected/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-kube-api-access-p8ng8\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.859192 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.859084 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.859192 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.859143 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-telemeter-client-tls\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.859192 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.859171 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-serving-certs-ca-bundle\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.859886 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.859823 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-metrics-client-ca\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.859886 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.859854 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-serving-certs-ca-bundle\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.860119 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.860029 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.861600 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.861576 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.861686 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.861634 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-federate-client-tls\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.861736 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.861721 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-secret-telemeter-client\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.861812 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.861789 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-telemeter-client-tls\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.868280 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.868257 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ng8\" (UniqueName: \"kubernetes.io/projected/b65ba86c-a92d-4db7-9052-e2a40ba8f9ac-kube-api-access-p8ng8\") pod \"telemeter-client-75df96dc46-88sdg\" (UID: \"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac\") " pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:02.896136 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:02.896113 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" Aug 04 18:12:03.033515 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:03.033494 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75df96dc46-88sdg"] Aug 04 18:12:03.035566 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:12:03.035537 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65ba86c_a92d_4db7_9052_e2a40ba8f9ac.slice/crio-bff8b70d7f5725636e0fe1a2b1f6285a98b1d674aa95445e08cecaa28752ce74 WatchSource:0}: Error finding container bff8b70d7f5725636e0fe1a2b1f6285a98b1d674aa95445e08cecaa28752ce74: Status 404 returned error can't find the container with id bff8b70d7f5725636e0fe1a2b1f6285a98b1d674aa95445e08cecaa28752ce74 Aug 04 18:12:03.465214 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:03.465180 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:03.542702 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:03.542670 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" event={"ID":"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac","Type":"ContainerStarted","Data":"bff8b70d7f5725636e0fe1a2b1f6285a98b1d674aa95445e08cecaa28752ce74"} Aug 04 18:12:03.971978 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:03.971941 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:04.058418 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:04.058391 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:06.555020 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:06.554954 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" event={"ID":"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac","Type":"ContainerStarted","Data":"c740b5663adf83172c0fffbaff2cbc0033258c470ffa3429a6812048c2d52bb1"} Aug 04 18:12:06.776368 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:06.776323 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:06.862363 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:06.862279 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:07.559825 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:07.559796 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" event={"ID":"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac","Type":"ContainerStarted","Data":"f1e8f2bcc71cc6feff13d18cef262ec45adf818885655bc5933512e8685687ad"} Aug 04 18:12:07.560201 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:07.559846 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" event={"ID":"b65ba86c-a92d-4db7-9052-e2a40ba8f9ac","Type":"ContainerStarted","Data":"3a2b4b7699a1fbbf5085a884cbaa0004663f6d69e77bdfb51ad8ee94fe239e4c"} Aug 04 18:12:07.584548 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:07.584505 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-75df96dc46-88sdg" podStartSLOduration=1.8010506099999999 podStartE2EDuration="5.584486987s" totalImagesPullingTime="3.783436377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:12:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:07.582784949 +0000 UTC m=+84.834067509" watchObservedRunningTime="2026-08-04 18:12:07.584486987 +0000 UTC m=+84.835769536" Aug 04 18:12:07.599618 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:07.599593 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:07.698786 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:07.698760 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:08.085588 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:08.085556 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:08.185204 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:08.185176 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:10.478037 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:10.478005 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:10.651891 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:10.651850 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:10.777680 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:10.777601 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:10.881000 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:10.880949 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:12.906471 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:12.906437 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:13.000226 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.000188 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:13.315094 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.315059 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:13.424321 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.424285 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:13.515888 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.515855 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" podUID="e375a34f-0263-4345-9317-067d37d3a95c" containerName="registry" containerID="cri-o://fa87d3c4d00aa10d24e48116d1842e4ffb8001e5aa7d42f8c28f162e545d6548" gracePeriod=30 Aug 04 18:12:13.759603 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.759579 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:12:13.938631 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.938568 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e375a34f-0263-4345-9317-067d37d3a95c-ca-trust-extracted\") pod \"e375a34f-0263-4345-9317-067d37d3a95c\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " Aug 04 18:12:13.938631 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.938597 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-bound-sa-token\") pod \"e375a34f-0263-4345-9317-067d37d3a95c\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " Aug 04 18:12:13.938631 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.938619 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e375a34f-0263-4345-9317-067d37d3a95c-image-registry-private-configuration\") pod \"e375a34f-0263-4345-9317-067d37d3a95c\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " Aug 04 18:12:13.939135 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.938640 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e375a34f-0263-4345-9317-067d37d3a95c-installation-pull-secrets\") pod \"e375a34f-0263-4345-9317-067d37d3a95c\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " Aug 04 18:12:13.939135 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.938714 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e375a34f-0263-4345-9317-067d37d3a95c-trusted-ca\") pod \"e375a34f-0263-4345-9317-067d37d3a95c\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " Aug 04 18:12:13.939135 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.938742 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e375a34f-0263-4345-9317-067d37d3a95c-registry-certificates\") pod \"e375a34f-0263-4345-9317-067d37d3a95c\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " Aug 04 18:12:13.939135 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.938766 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsptg\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-kube-api-access-qsptg\") pod \"e375a34f-0263-4345-9317-067d37d3a95c\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " Aug 04 18:12:13.939135 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.938794 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls\") pod \"e375a34f-0263-4345-9317-067d37d3a95c\" (UID: \"e375a34f-0263-4345-9317-067d37d3a95c\") " Aug 04 18:12:13.939376 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.939248 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e375a34f-0263-4345-9317-067d37d3a95c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e375a34f-0263-4345-9317-067d37d3a95c" (UID: "e375a34f-0263-4345-9317-067d37d3a95c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:12:13.939376 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.939288 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e375a34f-0263-4345-9317-067d37d3a95c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e375a34f-0263-4345-9317-067d37d3a95c" (UID: "e375a34f-0263-4345-9317-067d37d3a95c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:12:13.941248 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.941218 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e375a34f-0263-4345-9317-067d37d3a95c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e375a34f-0263-4345-9317-067d37d3a95c" (UID: "e375a34f-0263-4345-9317-067d37d3a95c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:12:13.941365 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.941279 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e375a34f-0263-4345-9317-067d37d3a95c" (UID: "e375a34f-0263-4345-9317-067d37d3a95c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:12:13.941365 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.941322 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e375a34f-0263-4345-9317-067d37d3a95c" (UID: "e375a34f-0263-4345-9317-067d37d3a95c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:12:13.941457 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.941434 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e375a34f-0263-4345-9317-067d37d3a95c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e375a34f-0263-4345-9317-067d37d3a95c" (UID: "e375a34f-0263-4345-9317-067d37d3a95c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:12:13.941549 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.941526 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-kube-api-access-qsptg" (OuterVolumeSpecName: "kube-api-access-qsptg") pod "e375a34f-0263-4345-9317-067d37d3a95c" (UID: "e375a34f-0263-4345-9317-067d37d3a95c"). InnerVolumeSpecName "kube-api-access-qsptg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:12:13.947635 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:13.947613 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e375a34f-0263-4345-9317-067d37d3a95c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e375a34f-0263-4345-9317-067d37d3a95c" (UID: "e375a34f-0263-4345-9317-067d37d3a95c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:12:14.039498 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.039473 2526 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e375a34f-0263-4345-9317-067d37d3a95c-registry-certificates\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:12:14.039498 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.039495 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsptg\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-kube-api-access-qsptg\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:12:14.039616 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.039507 2526 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-registry-tls\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:12:14.039616 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.039516 2526 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e375a34f-0263-4345-9317-067d37d3a95c-ca-trust-extracted\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:12:14.039616 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.039524 2526 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e375a34f-0263-4345-9317-067d37d3a95c-bound-sa-token\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:12:14.039616 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.039533 2526 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e375a34f-0263-4345-9317-067d37d3a95c-image-registry-private-configuration\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:12:14.039616 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.039541 2526 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e375a34f-0263-4345-9317-067d37d3a95c-installation-pull-secrets\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:12:14.039616 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.039549 2526 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e375a34f-0263-4345-9317-067d37d3a95c-trusted-ca\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:12:14.389075 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.389041 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:14.584916 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.584873 2526 generic.go:356] "Generic (PLEG): container finished" podID="e375a34f-0263-4345-9317-067d37d3a95c" containerID="fa87d3c4d00aa10d24e48116d1842e4ffb8001e5aa7d42f8c28f162e545d6548" exitCode=0 Aug 04 18:12:14.585084 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.584931 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" Aug 04 18:12:14.585084 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.584960 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" event={"ID":"e375a34f-0263-4345-9317-067d37d3a95c","Type":"ContainerDied","Data":"fa87d3c4d00aa10d24e48116d1842e4ffb8001e5aa7d42f8c28f162e545d6548"} Aug 04 18:12:14.585084 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.585028 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4c467488-kwzr5" event={"ID":"e375a34f-0263-4345-9317-067d37d3a95c","Type":"ContainerDied","Data":"7b886ab14e90557df9e7688577be32d2793bf6051b2615cc11ead4804af6ab15"} Aug 04 18:12:14.585799 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.585776 2526 scope.go:117] "RemoveContainer" containerID="fa87d3c4d00aa10d24e48116d1842e4ffb8001e5aa7d42f8c28f162e545d6548" Aug 04 18:12:14.594469 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.594450 2526 scope.go:117] "RemoveContainer" containerID="fa87d3c4d00aa10d24e48116d1842e4ffb8001e5aa7d42f8c28f162e545d6548" Aug 04 18:12:14.594702 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:12:14.594684 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa87d3c4d00aa10d24e48116d1842e4ffb8001e5aa7d42f8c28f162e545d6548\": container with ID starting with fa87d3c4d00aa10d24e48116d1842e4ffb8001e5aa7d42f8c28f162e545d6548 not found: ID does not exist" containerID="fa87d3c4d00aa10d24e48116d1842e4ffb8001e5aa7d42f8c28f162e545d6548" Aug 04 18:12:14.594749 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.594712 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa87d3c4d00aa10d24e48116d1842e4ffb8001e5aa7d42f8c28f162e545d6548"} err="failed to get container status \"fa87d3c4d00aa10d24e48116d1842e4ffb8001e5aa7d42f8c28f162e545d6548\": rpc error: code = NotFound desc = could not find container \"fa87d3c4d00aa10d24e48116d1842e4ffb8001e5aa7d42f8c28f162e545d6548\": container with ID starting with fa87d3c4d00aa10d24e48116d1842e4ffb8001e5aa7d42f8c28f162e545d6548 not found: ID does not exist" Aug 04 18:12:14.607800 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.607780 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f4c467488-kwzr5"] Aug 04 18:12:14.614455 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:14.614437 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f4c467488-kwzr5"] Aug 04 18:12:15.179563 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:15.179524 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e375a34f-0263-4345-9317-067d37d3a95c" path="/var/lib/kubelet/pods/e375a34f-0263-4345-9317-067d37d3a95c/volumes" Aug 04 18:12:16.287947 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:16.287903 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:16.414323 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:16.414291 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:16.517868 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:16.517828 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:19.560956 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:19.560920 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:19.663282 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:19.663253 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:20.131008 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:20.130960 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:20.230774 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:20.230743 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:20.340227 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:20.340194 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:21.724419 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:21.724391 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:21.825247 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:21.825216 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:27.514416 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:27.514388 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qmq4s" Aug 04 18:12:36.775242 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:36.775203 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:36.856795 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:36.856768 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:56.012771 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:12:56.012731 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:13:13.323927 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:13:13.323891 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:13:13.424518 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:13:13.424478 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:13:16.355549 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:13:16.355514 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:13:16.481981 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:13:16.481940 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:13:25.278127 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:13:25.278091 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:13:25.377210 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:13:25.377182 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:13:27.382806 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:13:27.382762 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:13:27.484172 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:13:27.484131 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:13:27.555405 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:13:27.555378 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:13:36.735912 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:13:36.735834 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:13:36.875051 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:13:36.875013 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:13:36.976496 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:13:36.976467 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:13:47.338380 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:13:47.338340 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:14:18.329683 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:14:18.329646 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:14:21.822601 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:14:21.822567 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:14:47.616764 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:14:47.616724 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:14:47.716522 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:14:47.716487 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:15:04.142538 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:15:04.142506 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:15:04.201310 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:15:04.201275 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:15:43.044150 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:15:43.044110 2526 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 18:16:14.780938 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:14.780903 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:16.287388 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:16.287353 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:17.695507 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:17.695467 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:17.796136 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:17.796104 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:18.594364 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:18.594328 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:18.694144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:18.694103 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:19.514507 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:19.514468 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:19.614806 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:19.614773 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:20.242561 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:20.242526 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:20.352545 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:20.352515 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:21.983829 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:21.983787 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:22.373695 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:22.373605 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:25.353639 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:25.353607 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:25.423639 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:25.423595 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:27.788740 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:27.788696 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:27.906159 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:27.906121 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:30.173545 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.173515 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:30.458295 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.458266 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zhdnr"] Aug 04 18:16:30.458494 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.458483 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e375a34f-0263-4345-9317-067d37d3a95c" containerName="registry" Aug 04 18:16:30.458545 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.458496 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="e375a34f-0263-4345-9317-067d37d3a95c" containerName="registry" Aug 04 18:16:30.458579 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.458553 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="e375a34f-0263-4345-9317-067d37d3a95c" containerName="registry" Aug 04 18:16:30.461374 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.461354 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-zhdnr" Aug 04 18:16:30.463590 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.463566 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-x54bg\"" Aug 04 18:16:30.463677 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.463629 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 04 18:16:30.463735 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.463719 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 04 18:16:30.470767 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.470747 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zhdnr"] Aug 04 18:16:30.529063 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.529034 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckt65\" (UniqueName: \"kubernetes.io/projected/fe115082-e95f-4e38-a228-2333b69d5280-kube-api-access-ckt65\") pod \"cert-manager-55b4d954d5-zhdnr\" (UID: \"fe115082-e95f-4e38-a228-2333b69d5280\") " pod="cert-manager/cert-manager-55b4d954d5-zhdnr" Aug 04 18:16:30.529205 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.529091 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe115082-e95f-4e38-a228-2333b69d5280-bound-sa-token\") pod \"cert-manager-55b4d954d5-zhdnr\" (UID: \"fe115082-e95f-4e38-a228-2333b69d5280\") " pod="cert-manager/cert-manager-55b4d954d5-zhdnr" Aug 04 18:16:30.629482 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.629390 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe115082-e95f-4e38-a228-2333b69d5280-bound-sa-token\") pod \"cert-manager-55b4d954d5-zhdnr\" (UID: \"fe115082-e95f-4e38-a228-2333b69d5280\") " pod="cert-manager/cert-manager-55b4d954d5-zhdnr" Aug 04 18:16:30.629640 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.629528 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckt65\" (UniqueName: \"kubernetes.io/projected/fe115082-e95f-4e38-a228-2333b69d5280-kube-api-access-ckt65\") pod \"cert-manager-55b4d954d5-zhdnr\" (UID: \"fe115082-e95f-4e38-a228-2333b69d5280\") " pod="cert-manager/cert-manager-55b4d954d5-zhdnr" Aug 04 18:16:30.638901 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.638873 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe115082-e95f-4e38-a228-2333b69d5280-bound-sa-token\") pod \"cert-manager-55b4d954d5-zhdnr\" (UID: \"fe115082-e95f-4e38-a228-2333b69d5280\") " pod="cert-manager/cert-manager-55b4d954d5-zhdnr" Aug 04 18:16:30.639187 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.639169 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckt65\" (UniqueName: \"kubernetes.io/projected/fe115082-e95f-4e38-a228-2333b69d5280-kube-api-access-ckt65\") pod \"cert-manager-55b4d954d5-zhdnr\" (UID: \"fe115082-e95f-4e38-a228-2333b69d5280\") " pod="cert-manager/cert-manager-55b4d954d5-zhdnr" Aug 04 18:16:30.771315 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.771262 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-zhdnr" Aug 04 18:16:30.890555 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.890438 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zhdnr"] Aug 04 18:16:30.893107 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:16:30.893081 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe115082_e95f_4e38_a228_2333b69d5280.slice/crio-88dda5858a51614333cac9828ea4c4ad3acd875a05faf7897543287442627472 WatchSource:0}: Error finding container 88dda5858a51614333cac9828ea4c4ad3acd875a05faf7897543287442627472: Status 404 returned error can't find the container with id 88dda5858a51614333cac9828ea4c4ad3acd875a05faf7897543287442627472 Aug 04 18:16:30.895154 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:30.895138 2526 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:16:31.399176 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:31.399138 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-zhdnr" event={"ID":"fe115082-e95f-4e38-a228-2333b69d5280","Type":"ContainerStarted","Data":"88dda5858a51614333cac9828ea4c4ad3acd875a05faf7897543287442627472"} Aug 04 18:16:34.412504 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:34.412467 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-zhdnr" event={"ID":"fe115082-e95f-4e38-a228-2333b69d5280","Type":"ContainerStarted","Data":"21ab7b35802d0d1351b117de2b44b9fd262b04268a659cee39ba08c2d7b54e92"} Aug 04 18:16:34.432038 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:34.431996 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-zhdnr" podStartSLOduration=1.405021404 podStartE2EDuration="4.431980186s" totalImagesPullingTime="3.026958782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:16:34.430917144 +0000 UTC m=+351.682199693" watchObservedRunningTime="2026-08-04 18:16:34.431980186 +0000 UTC m=+351.683262728" Aug 04 18:16:35.831149 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:35.831070 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:36.461188 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:36.461152 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:38.011712 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:38.011679 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jw8gm"] Aug 04 18:16:38.014587 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:38.014571 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jw8gm" Aug 04 18:16:38.017145 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:38.017124 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 18:16:38.018139 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:38.018122 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-4hrjd\"" Aug 04 18:16:38.018139 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:38.018137 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 18:16:38.024428 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:38.024410 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jw8gm"] Aug 04 18:16:38.083796 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:38.083772 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a79ff6c-01b0-4585-ab7c-97af9c6e0327-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jw8gm\" (UID: \"9a79ff6c-01b0-4585-ab7c-97af9c6e0327\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jw8gm" Aug 04 18:16:38.083926 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:38.083854 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb7zk\" (UniqueName: \"kubernetes.io/projected/9a79ff6c-01b0-4585-ab7c-97af9c6e0327-kube-api-access-pb7zk\") pod \"openshift-lws-operator-fd8ccff4c-jw8gm\" (UID: \"9a79ff6c-01b0-4585-ab7c-97af9c6e0327\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jw8gm" Aug 04 18:16:38.184524 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:38.184497 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a79ff6c-01b0-4585-ab7c-97af9c6e0327-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jw8gm\" (UID: \"9a79ff6c-01b0-4585-ab7c-97af9c6e0327\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jw8gm" Aug 04 18:16:38.184668 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:38.184546 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb7zk\" (UniqueName: \"kubernetes.io/projected/9a79ff6c-01b0-4585-ab7c-97af9c6e0327-kube-api-access-pb7zk\") pod \"openshift-lws-operator-fd8ccff4c-jw8gm\" (UID: \"9a79ff6c-01b0-4585-ab7c-97af9c6e0327\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jw8gm" Aug 04 18:16:38.184897 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:38.184877 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a79ff6c-01b0-4585-ab7c-97af9c6e0327-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jw8gm\" (UID: \"9a79ff6c-01b0-4585-ab7c-97af9c6e0327\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jw8gm" Aug 04 18:16:38.193874 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:38.193852 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb7zk\" (UniqueName: \"kubernetes.io/projected/9a79ff6c-01b0-4585-ab7c-97af9c6e0327-kube-api-access-pb7zk\") pod \"openshift-lws-operator-fd8ccff4c-jw8gm\" (UID: \"9a79ff6c-01b0-4585-ab7c-97af9c6e0327\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jw8gm" Aug 04 18:16:38.322817 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:38.322769 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jw8gm" Aug 04 18:16:38.447503 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:38.447472 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jw8gm"] Aug 04 18:16:38.450584 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:16:38.450558 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a79ff6c_01b0_4585_ab7c_97af9c6e0327.slice/crio-a3d17839797548d7ed769d49d78ea4ce7c802195fccde340a1201cf8869bb508 WatchSource:0}: Error finding container a3d17839797548d7ed769d49d78ea4ce7c802195fccde340a1201cf8869bb508: Status 404 returned error can't find the container with id a3d17839797548d7ed769d49d78ea4ce7c802195fccde340a1201cf8869bb508 Aug 04 18:16:39.435665 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:39.435620 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jw8gm" event={"ID":"9a79ff6c-01b0-4585-ab7c-97af9c6e0327","Type":"ContainerStarted","Data":"a3d17839797548d7ed769d49d78ea4ce7c802195fccde340a1201cf8869bb508"} Aug 04 18:16:42.447860 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:42.447827 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jw8gm" event={"ID":"9a79ff6c-01b0-4585-ab7c-97af9c6e0327","Type":"ContainerStarted","Data":"d5dc28e05c0ba2614f904321a57ced4edd1b962ff57eb2f3725fd42640413f0a"} Aug 04 18:16:42.470214 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:42.470168 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jw8gm" podStartSLOduration=2.45550909 podStartE2EDuration="5.470154153s" totalImagesPullingTime="3.014645063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:16:42.469656006 +0000 UTC m=+359.720938546" watchObservedRunningTime="2026-08-04 18:16:42.470154153 +0000 UTC m=+359.721436701" Aug 04 18:16:43.307415 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:43.307378 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:43.407826 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:43.407794 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:45.546290 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:45.546259 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:59.716185 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:59.716142 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:59.818429 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:59.815720 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:16:59.977395 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:16:59.977317 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:00.093027 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:00.092985 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:00.358325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:00.358240 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:01.378597 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.378562 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:01.483908 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.483871 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:01.542217 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.542179 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg"] Aug 04 18:17:01.544498 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.544482 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.549118 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.549095 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 18:17:01.549241 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.549094 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 18:17:01.549469 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.549448 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 18:17:01.549469 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.549462 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qb6pd\"" Aug 04 18:17:01.557416 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.557400 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 18:17:01.561271 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.561250 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg"] Aug 04 18:17:01.643455 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.643374 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb341c5b-c651-4e86-a138-f9d2b0d747cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-s48rg\" (UID: \"eb341c5b-c651-4e86-a138-f9d2b0d747cf\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.643455 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.643406 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j4fc\" (UniqueName: \"kubernetes.io/projected/eb341c5b-c651-4e86-a138-f9d2b0d747cf-kube-api-access-4j4fc\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-s48rg\" (UID: \"eb341c5b-c651-4e86-a138-f9d2b0d747cf\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.643455 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.643427 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eb341c5b-c651-4e86-a138-f9d2b0d747cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-s48rg\" (UID: \"eb341c5b-c651-4e86-a138-f9d2b0d747cf\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.643664 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.643484 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eb341c5b-c651-4e86-a138-f9d2b0d747cf-manifests\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-s48rg\" (UID: \"eb341c5b-c651-4e86-a138-f9d2b0d747cf\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.643664 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.643511 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb341c5b-c651-4e86-a138-f9d2b0d747cf-tmp\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-s48rg\" (UID: \"eb341c5b-c651-4e86-a138-f9d2b0d747cf\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.744698 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.744664 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb341c5b-c651-4e86-a138-f9d2b0d747cf-tmp\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-s48rg\" (UID: \"eb341c5b-c651-4e86-a138-f9d2b0d747cf\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.744870 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.744718 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb341c5b-c651-4e86-a138-f9d2b0d747cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-s48rg\" (UID: \"eb341c5b-c651-4e86-a138-f9d2b0d747cf\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.744870 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.744744 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4fc\" (UniqueName: \"kubernetes.io/projected/eb341c5b-c651-4e86-a138-f9d2b0d747cf-kube-api-access-4j4fc\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-s48rg\" (UID: \"eb341c5b-c651-4e86-a138-f9d2b0d747cf\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.744870 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.744774 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eb341c5b-c651-4e86-a138-f9d2b0d747cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-s48rg\" (UID: \"eb341c5b-c651-4e86-a138-f9d2b0d747cf\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.744870 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.744816 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eb341c5b-c651-4e86-a138-f9d2b0d747cf-manifests\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-s48rg\" (UID: \"eb341c5b-c651-4e86-a138-f9d2b0d747cf\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.745282 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.745253 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eb341c5b-c651-4e86-a138-f9d2b0d747cf-manifests\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-s48rg\" (UID: \"eb341c5b-c651-4e86-a138-f9d2b0d747cf\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.747327 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.747304 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb341c5b-c651-4e86-a138-f9d2b0d747cf-tmp\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-s48rg\" (UID: \"eb341c5b-c651-4e86-a138-f9d2b0d747cf\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.747327 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.747318 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eb341c5b-c651-4e86-a138-f9d2b0d747cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-s48rg\" (UID: \"eb341c5b-c651-4e86-a138-f9d2b0d747cf\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.747450 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.747349 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb341c5b-c651-4e86-a138-f9d2b0d747cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-s48rg\" (UID: \"eb341c5b-c651-4e86-a138-f9d2b0d747cf\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.755012 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.754991 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j4fc\" (UniqueName: \"kubernetes.io/projected/eb341c5b-c651-4e86-a138-f9d2b0d747cf-kube-api-access-4j4fc\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-s48rg\" (UID: \"eb341c5b-c651-4e86-a138-f9d2b0d747cf\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.758090 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.758053 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:01.854112 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.854080 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:01.875879 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.875833 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:01.988622 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:01.988597 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg"] Aug 04 18:17:01.991299 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:17:01.991274 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb341c5b_c651_4e86_a138_f9d2b0d747cf.slice/crio-822c5232ce68280815b1e591d761baf3545669cb7f5b98f754a910e5b175a640 WatchSource:0}: Error finding container 822c5232ce68280815b1e591d761baf3545669cb7f5b98f754a910e5b175a640: Status 404 returned error can't find the container with id 822c5232ce68280815b1e591d761baf3545669cb7f5b98f754a910e5b175a640 Aug 04 18:17:02.512764 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:02.512730 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" event={"ID":"eb341c5b-c651-4e86-a138-f9d2b0d747cf","Type":"ContainerStarted","Data":"822c5232ce68280815b1e591d761baf3545669cb7f5b98f754a910e5b175a640"} Aug 04 18:17:02.904060 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:02.903916 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:03.083928 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:03.083891 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:05.524531 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:05.524499 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" event={"ID":"eb341c5b-c651-4e86-a138-f9d2b0d747cf","Type":"ContainerStarted","Data":"6fccbb42b00dcc283d40930baeea4c98a1af7e193967cc9e80ce28aaf004327e"} Aug 04 18:17:06.529774 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:06.529694 2526 generic.go:356] "Generic (PLEG): container finished" podID="eb341c5b-c651-4e86-a138-f9d2b0d747cf" containerID="6fccbb42b00dcc283d40930baeea4c98a1af7e193967cc9e80ce28aaf004327e" exitCode=0 Aug 04 18:17:06.530151 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:06.529784 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" event={"ID":"eb341c5b-c651-4e86-a138-f9d2b0d747cf","Type":"ContainerDied","Data":"6fccbb42b00dcc283d40930baeea4c98a1af7e193967cc9e80ce28aaf004327e"} Aug 04 18:17:07.534826 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:07.534791 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" event={"ID":"eb341c5b-c651-4e86-a138-f9d2b0d747cf","Type":"ContainerStarted","Data":"1f2accf045c1dfbd49dcb7ccf70f4d02d7ea84f67b1e6a00ecbf92cf993e63a2"} Aug 04 18:17:07.535250 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:07.534923 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:07.567788 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:07.567732 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" podStartSLOduration=1.8647383720000001 podStartE2EDuration="6.567711638s" totalImagesPullingTime="3.790447866s" totalInitContainerRuntime="912.5254ms" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:07.565204614 +0000 UTC m=+384.816487160" watchObservedRunningTime="2026-08-04 18:17:07.567711638 +0000 UTC m=+384.818994186" Aug 04 18:17:18.541201 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:18.541171 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:17:21.862806 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:21.862766 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:22.066031 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.065997 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:22.273433 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.273395 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:22.278392 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.278368 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8489764dcd-7bclw"] Aug 04 18:17:22.287458 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.287434 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8489764dcd-7bclw" Aug 04 18:17:22.293550 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.293531 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 04 18:17:22.293680 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.293662 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-47hbm\"" Aug 04 18:17:22.294432 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.294410 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 18:17:22.294566 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.294433 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 18:17:22.298248 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.298229 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 18:17:22.306232 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.306210 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8489764dcd-7bclw"] Aug 04 18:17:22.393102 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.393068 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c12d64-b08e-4ca2-af73-c51e416fdd13-tls-certs\") pod \"kube-auth-proxy-8489764dcd-7bclw\" (UID: \"32c12d64-b08e-4ca2-af73-c51e416fdd13\") " pod="openshift-ingress/kube-auth-proxy-8489764dcd-7bclw" Aug 04 18:17:22.393261 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.393115 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2xhl\" (UniqueName: \"kubernetes.io/projected/32c12d64-b08e-4ca2-af73-c51e416fdd13-kube-api-access-r2xhl\") pod \"kube-auth-proxy-8489764dcd-7bclw\" (UID: \"32c12d64-b08e-4ca2-af73-c51e416fdd13\") " pod="openshift-ingress/kube-auth-proxy-8489764dcd-7bclw" Aug 04 18:17:22.393261 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.393182 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32c12d64-b08e-4ca2-af73-c51e416fdd13-tmp\") pod \"kube-auth-proxy-8489764dcd-7bclw\" (UID: \"32c12d64-b08e-4ca2-af73-c51e416fdd13\") " pod="openshift-ingress/kube-auth-proxy-8489764dcd-7bclw" Aug 04 18:17:22.494099 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.494072 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c12d64-b08e-4ca2-af73-c51e416fdd13-tls-certs\") pod \"kube-auth-proxy-8489764dcd-7bclw\" (UID: \"32c12d64-b08e-4ca2-af73-c51e416fdd13\") " pod="openshift-ingress/kube-auth-proxy-8489764dcd-7bclw" Aug 04 18:17:22.494268 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.494113 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2xhl\" (UniqueName: \"kubernetes.io/projected/32c12d64-b08e-4ca2-af73-c51e416fdd13-kube-api-access-r2xhl\") pod \"kube-auth-proxy-8489764dcd-7bclw\" (UID: \"32c12d64-b08e-4ca2-af73-c51e416fdd13\") " pod="openshift-ingress/kube-auth-proxy-8489764dcd-7bclw" Aug 04 18:17:22.494268 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.494136 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32c12d64-b08e-4ca2-af73-c51e416fdd13-tmp\") pod \"kube-auth-proxy-8489764dcd-7bclw\" (UID: \"32c12d64-b08e-4ca2-af73-c51e416fdd13\") " pod="openshift-ingress/kube-auth-proxy-8489764dcd-7bclw" Aug 04 18:17:22.496418 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.496385 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32c12d64-b08e-4ca2-af73-c51e416fdd13-tmp\") pod \"kube-auth-proxy-8489764dcd-7bclw\" (UID: \"32c12d64-b08e-4ca2-af73-c51e416fdd13\") " pod="openshift-ingress/kube-auth-proxy-8489764dcd-7bclw" Aug 04 18:17:22.496586 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.496565 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c12d64-b08e-4ca2-af73-c51e416fdd13-tls-certs\") pod \"kube-auth-proxy-8489764dcd-7bclw\" (UID: \"32c12d64-b08e-4ca2-af73-c51e416fdd13\") " pod="openshift-ingress/kube-auth-proxy-8489764dcd-7bclw" Aug 04 18:17:22.549388 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.549329 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2xhl\" (UniqueName: \"kubernetes.io/projected/32c12d64-b08e-4ca2-af73-c51e416fdd13-kube-api-access-r2xhl\") pod \"kube-auth-proxy-8489764dcd-7bclw\" (UID: \"32c12d64-b08e-4ca2-af73-c51e416fdd13\") " pod="openshift-ingress/kube-auth-proxy-8489764dcd-7bclw" Aug 04 18:17:22.588986 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.588923 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:22.599929 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.599906 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8489764dcd-7bclw" Aug 04 18:17:22.744413 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.744384 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8489764dcd-7bclw"] Aug 04 18:17:22.747496 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:17:22.747467 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c12d64_b08e_4ca2_af73_c51e416fdd13.slice/crio-08c000e4506e475e21b97ce5c0998c758df5708df97857c7bbec6df0a9cd6b7e WatchSource:0}: Error finding container 08c000e4506e475e21b97ce5c0998c758df5708df97857c7bbec6df0a9cd6b7e: Status 404 returned error can't find the container with id 08c000e4506e475e21b97ce5c0998c758df5708df97857c7bbec6df0a9cd6b7e Aug 04 18:17:22.942134 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.942059 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fdvj6"] Aug 04 18:17:22.946221 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.946204 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" Aug 04 18:17:22.948746 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.948721 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-c29d5\"" Aug 04 18:17:22.949183 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.949164 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Aug 04 18:17:22.963187 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:22.963162 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fdvj6"] Aug 04 18:17:23.041021 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:23.040948 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:23.097917 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:23.097881 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk4hb\" (UniqueName: \"kubernetes.io/projected/c6cc8554-0e4a-4b65-af23-661da2fcf0f3-kube-api-access-mk4hb\") pod \"odh-model-controller-7998c67dc4-fdvj6\" (UID: \"c6cc8554-0e4a-4b65-af23-661da2fcf0f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" Aug 04 18:17:23.098127 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:23.097930 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6cc8554-0e4a-4b65-af23-661da2fcf0f3-cert\") pod \"odh-model-controller-7998c67dc4-fdvj6\" (UID: \"c6cc8554-0e4a-4b65-af23-661da2fcf0f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" Aug 04 18:17:23.158481 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:23.158449 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:23.198819 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:23.198794 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6cc8554-0e4a-4b65-af23-661da2fcf0f3-cert\") pod \"odh-model-controller-7998c67dc4-fdvj6\" (UID: \"c6cc8554-0e4a-4b65-af23-661da2fcf0f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" Aug 04 18:17:23.198987 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:23.198871 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk4hb\" (UniqueName: \"kubernetes.io/projected/c6cc8554-0e4a-4b65-af23-661da2fcf0f3-kube-api-access-mk4hb\") pod \"odh-model-controller-7998c67dc4-fdvj6\" (UID: \"c6cc8554-0e4a-4b65-af23-661da2fcf0f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" Aug 04 18:17:23.201544 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:23.201523 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6cc8554-0e4a-4b65-af23-661da2fcf0f3-cert\") pod \"odh-model-controller-7998c67dc4-fdvj6\" (UID: \"c6cc8554-0e4a-4b65-af23-661da2fcf0f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" Aug 04 18:17:23.210284 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:23.210261 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk4hb\" (UniqueName: \"kubernetes.io/projected/c6cc8554-0e4a-4b65-af23-661da2fcf0f3-kube-api-access-mk4hb\") pod \"odh-model-controller-7998c67dc4-fdvj6\" (UID: \"c6cc8554-0e4a-4b65-af23-661da2fcf0f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" Aug 04 18:17:23.255970 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:23.255946 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" Aug 04 18:17:23.256699 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:23.256672 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:23.362794 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:23.362705 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:23.400485 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:23.400459 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fdvj6"] Aug 04 18:17:23.402249 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:17:23.402227 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6cc8554_0e4a_4b65_af23_661da2fcf0f3.slice/crio-baa8177bdf7a881812fdf72e56e798fb3d3a1b7a30256e54d6bce33a2abd5315 WatchSource:0}: Error finding container baa8177bdf7a881812fdf72e56e798fb3d3a1b7a30256e54d6bce33a2abd5315: Status 404 returned error can't find the container with id baa8177bdf7a881812fdf72e56e798fb3d3a1b7a30256e54d6bce33a2abd5315 Aug 04 18:17:23.590042 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:23.589932 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" event={"ID":"c6cc8554-0e4a-4b65-af23-661da2fcf0f3","Type":"ContainerStarted","Data":"baa8177bdf7a881812fdf72e56e798fb3d3a1b7a30256e54d6bce33a2abd5315"} Aug 04 18:17:23.591428 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:23.591400 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8489764dcd-7bclw" event={"ID":"32c12d64-b08e-4ca2-af73-c51e416fdd13","Type":"ContainerStarted","Data":"08c000e4506e475e21b97ce5c0998c758df5708df97857c7bbec6df0a9cd6b7e"} Aug 04 18:17:24.195146 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.194951 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:24.236403 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.235282 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-gkwpb"] Aug 04 18:17:24.238763 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.238742 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-gkwpb" Aug 04 18:17:24.243035 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.243017 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Aug 04 18:17:24.282753 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.282713 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-gkwpb"] Aug 04 18:17:24.410400 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.409791 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2960465c-dc9e-49be-a178-fbeb55d9f1d2-tls-certs\") pod \"model-serving-api-54d89bd5fc-gkwpb\" (UID: \"2960465c-dc9e-49be-a178-fbeb55d9f1d2\") " pod="opendatahub/model-serving-api-54d89bd5fc-gkwpb" Aug 04 18:17:24.410400 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.409853 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx2lv\" (UniqueName: \"kubernetes.io/projected/2960465c-dc9e-49be-a178-fbeb55d9f1d2-kube-api-access-kx2lv\") pod \"model-serving-api-54d89bd5fc-gkwpb\" (UID: \"2960465c-dc9e-49be-a178-fbeb55d9f1d2\") " pod="opendatahub/model-serving-api-54d89bd5fc-gkwpb" Aug 04 18:17:24.484835 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.483489 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5"] Aug 04 18:17:24.487136 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.487077 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.493878 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.493852 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 04 18:17:24.494321 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.494235 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 04 18:17:24.494321 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.494263 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 18:17:24.494462 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.494411 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 04 18:17:24.494520 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.494473 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tmws8\"" Aug 04 18:17:24.509519 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.509495 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5"] Aug 04 18:17:24.511184 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.511162 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2960465c-dc9e-49be-a178-fbeb55d9f1d2-tls-certs\") pod \"model-serving-api-54d89bd5fc-gkwpb\" (UID: \"2960465c-dc9e-49be-a178-fbeb55d9f1d2\") " pod="opendatahub/model-serving-api-54d89bd5fc-gkwpb" Aug 04 18:17:24.511262 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.511216 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx2lv\" (UniqueName: \"kubernetes.io/projected/2960465c-dc9e-49be-a178-fbeb55d9f1d2-kube-api-access-kx2lv\") pod \"model-serving-api-54d89bd5fc-gkwpb\" (UID: \"2960465c-dc9e-49be-a178-fbeb55d9f1d2\") " pod="opendatahub/model-serving-api-54d89bd5fc-gkwpb" Aug 04 18:17:24.511335 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:17:24.511316 2526 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Aug 04 18:17:24.511446 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:17:24.511400 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2960465c-dc9e-49be-a178-fbeb55d9f1d2-tls-certs podName:2960465c-dc9e-49be-a178-fbeb55d9f1d2 nodeName:}" failed. No retries permitted until 2026-08-04 18:17:25.011380353 +0000 UTC m=+402.262662878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2960465c-dc9e-49be-a178-fbeb55d9f1d2-tls-certs") pod "model-serving-api-54d89bd5fc-gkwpb" (UID: "2960465c-dc9e-49be-a178-fbeb55d9f1d2") : secret "model-serving-api-tls" not found Aug 04 18:17:24.516932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.516813 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:24.526516 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.526493 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx2lv\" (UniqueName: \"kubernetes.io/projected/2960465c-dc9e-49be-a178-fbeb55d9f1d2-kube-api-access-kx2lv\") pod \"model-serving-api-54d89bd5fc-gkwpb\" (UID: \"2960465c-dc9e-49be-a178-fbeb55d9f1d2\") " pod="opendatahub/model-serving-api-54d89bd5fc-gkwpb" Aug 04 18:17:24.612600 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.612566 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.612786 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.612626 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.612786 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.612679 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.612786 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.612752 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.612786 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.612781 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.613029 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.612815 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.613029 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.612838 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nzpq\" (UniqueName: \"kubernetes.io/projected/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-kube-api-access-8nzpq\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.616161 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.616130 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:24.713734 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.713675 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.713734 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.713723 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.715245 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.715165 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.715245 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.713753 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.715245 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.715231 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nzpq\" (UniqueName: \"kubernetes.io/projected/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-kube-api-access-8nzpq\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.715462 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.715307 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.715462 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.715347 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.715462 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.715382 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.718611 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.718590 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.718711 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.718644 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.721075 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.721050 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.722540 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.722514 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.730018 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.729993 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nzpq\" (UniqueName: \"kubernetes.io/projected/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-kube-api-access-8nzpq\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.731393 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.731369 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-gndv5\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:24.800473 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:24.800399 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:25.018862 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:25.018824 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2960465c-dc9e-49be-a178-fbeb55d9f1d2-tls-certs\") pod \"model-serving-api-54d89bd5fc-gkwpb\" (UID: \"2960465c-dc9e-49be-a178-fbeb55d9f1d2\") " pod="opendatahub/model-serving-api-54d89bd5fc-gkwpb" Aug 04 18:17:25.023227 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:25.023197 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2960465c-dc9e-49be-a178-fbeb55d9f1d2-tls-certs\") pod \"model-serving-api-54d89bd5fc-gkwpb\" (UID: \"2960465c-dc9e-49be-a178-fbeb55d9f1d2\") " pod="opendatahub/model-serving-api-54d89bd5fc-gkwpb" Aug 04 18:17:25.044005 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:25.043951 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:25.155674 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:25.155601 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-gkwpb" Aug 04 18:17:25.956536 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:25.956497 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:27.705206 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:27.705171 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5"] Aug 04 18:17:27.709683 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:17:27.709652 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781daeeb_2bd8_4741_b4c4_c7222bc4a7d8.slice/crio-867efab6a7f0c578cd6c4a05bdaeca570f769adafd3234bea52c158c6fcddfa0 WatchSource:0}: Error finding container 867efab6a7f0c578cd6c4a05bdaeca570f769adafd3234bea52c158c6fcddfa0: Status 404 returned error can't find the container with id 867efab6a7f0c578cd6c4a05bdaeca570f769adafd3234bea52c158c6fcddfa0 Aug 04 18:17:27.720477 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:27.720459 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-gkwpb"] Aug 04 18:17:27.724492 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:17:27.724452 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2960465c_dc9e_49be_a178_fbeb55d9f1d2.slice/crio-465897e9270c0c51e395fb7a2d42d3d2a12cf9d07b5624254e61f653ebdd5d8b WatchSource:0}: Error finding container 465897e9270c0c51e395fb7a2d42d3d2a12cf9d07b5624254e61f653ebdd5d8b: Status 404 returned error can't find the container with id 465897e9270c0c51e395fb7a2d42d3d2a12cf9d07b5624254e61f653ebdd5d8b Aug 04 18:17:28.619910 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:28.619839 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" event={"ID":"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8","Type":"ContainerStarted","Data":"867efab6a7f0c578cd6c4a05bdaeca570f769adafd3234bea52c158c6fcddfa0"} Aug 04 18:17:28.622623 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:28.622514 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-gkwpb" event={"ID":"2960465c-dc9e-49be-a178-fbeb55d9f1d2","Type":"ContainerStarted","Data":"465897e9270c0c51e395fb7a2d42d3d2a12cf9d07b5624254e61f653ebdd5d8b"} Aug 04 18:17:28.624380 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:28.624353 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" event={"ID":"c6cc8554-0e4a-4b65-af23-661da2fcf0f3","Type":"ContainerStarted","Data":"be1668b8b7291d0a5efb1bd6960bd58a9e4b4e1cbc6eea4428808c11d96938a0"} Aug 04 18:17:28.624524 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:28.624504 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" Aug 04 18:17:28.626021 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:28.625998 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8489764dcd-7bclw" event={"ID":"32c12d64-b08e-4ca2-af73-c51e416fdd13","Type":"ContainerStarted","Data":"9009bebbfbbbd220e98d5f46b5fe648d002d6b2a3cb40c01042d081e93838979"} Aug 04 18:17:28.653748 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:28.653053 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" podStartSLOduration=2.466024021 podStartE2EDuration="6.653034254s" totalImagesPullingTime="4.187010233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:28.652772395 +0000 UTC m=+405.904054942" watchObservedRunningTime="2026-08-04 18:17:28.653034254 +0000 UTC m=+405.904316802" Aug 04 18:17:30.661821 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:30.661774 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:17:30.662148 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:30.661857 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:17:31.639827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:31.639787 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-gkwpb" event={"ID":"2960465c-dc9e-49be-a178-fbeb55d9f1d2","Type":"ContainerStarted","Data":"e2119b2d62ee2fa6b5974369f5553c123d948d72c964dddb110d93973ec2a493"} Aug 04 18:17:31.640027 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:31.639860 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-gkwpb" Aug 04 18:17:31.641355 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:31.641329 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" event={"ID":"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8","Type":"ContainerStarted","Data":"e1c416a5b90af0b28ce92d36c1ef59abbe2c30ae70ac9af0068954f2da76b129"} Aug 04 18:17:31.641526 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:31.641512 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:31.643034 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:31.643010 2526 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-gndv5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 18:17:31.643108 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:31.643058 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" podUID="781daeeb-2bd8-4741-b4c4-c7222bc4a7d8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:17:31.660774 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:31.660715 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8489764dcd-7bclw" podStartSLOduration=4.865205745 podStartE2EDuration="9.660697414s" totalImagesPullingTime="4.795491669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:28.677753143 +0000 UTC m=+405.929035691" watchObservedRunningTime="2026-08-04 18:17:31.660697414 +0000 UTC m=+408.911979962" Aug 04 18:17:31.689466 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:31.689424 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" podStartSLOduration=4.7397373510000005 podStartE2EDuration="7.68940776s" totalImagesPullingTime="2.949670409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:31.688154289 +0000 UTC m=+408.939436836" watchObservedRunningTime="2026-08-04 18:17:31.68940776 +0000 UTC m=+408.940690309" Aug 04 18:17:31.689753 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:31.689689 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-gkwpb" podStartSLOduration=4.757449895 podStartE2EDuration="7.68968248s" totalImagesPullingTime="2.932232585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:31.660868899 +0000 UTC m=+408.912151446" watchObservedRunningTime="2026-08-04 18:17:31.68968248 +0000 UTC m=+408.940965028" Aug 04 18:17:31.867102 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:31.867067 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:32.011280 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.011250 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:32.645328 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.645290 2526 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-gndv5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 18:17:32.645490 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.645342 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" podUID="781daeeb-2bd8-4741-b4c4-c7222bc4a7d8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:17:32.780806 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.780775 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv"] Aug 04 18:17:32.784378 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.784354 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.788017 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.787997 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8hnzm\"" Aug 04 18:17:32.819784 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.819761 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv"] Aug 04 18:17:32.887714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.887685 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.887714 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.887717 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.887911 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.887740 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.887911 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.887787 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.887911 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.887815 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssccw\" (UniqueName: \"kubernetes.io/projected/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-kube-api-access-ssccw\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.887911 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.887850 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.887911 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.887899 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.888204 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.887934 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.888204 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.887950 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.889768 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.889745 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:32.989259 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.989233 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.989426 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.989293 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.989426 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.989347 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssccw\" (UniqueName: \"kubernetes.io/projected/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-kube-api-access-ssccw\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.989426 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.989393 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.989590 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.989424 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.989590 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.989454 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.989590 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.989478 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.989590 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.989507 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.989590 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.989545 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.989835 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.989625 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.989835 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.989749 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.990111 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.990090 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.990696 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.990671 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.990810 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.990781 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.991841 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.991818 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:32.992111 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:32.992094 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:33.004986 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:33.004945 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:33.005694 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:33.005667 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssccw\" (UniqueName: \"kubernetes.io/projected/8f3e6bd2-9d7a-4354-8775-1a0f0292891c-kube-api-access-ssccw\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv\" (UID: \"8f3e6bd2-9d7a-4354-8775-1a0f0292891c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:33.095298 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:33.095264 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:33.127025 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:33.126959 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:33.291619 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:33.291549 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv"] Aug 04 18:17:33.294654 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:17:33.294629 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3e6bd2_9d7a_4354_8775_1a0f0292891c.slice/crio-f09dd458653c6e4d8641d631af022b05a0af8a9e4435e30b857e789b4172ec47 WatchSource:0}: Error finding container f09dd458653c6e4d8641d631af022b05a0af8a9e4435e30b857e789b4172ec47: Status 404 returned error can't find the container with id f09dd458653c6e4d8641d631af022b05a0af8a9e4435e30b857e789b4172ec47 Aug 04 18:17:33.333005 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:33.332950 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:33.649387 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:33.649299 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" event={"ID":"8f3e6bd2-9d7a-4354-8775-1a0f0292891c","Type":"ContainerStarted","Data":"f09dd458653c6e4d8641d631af022b05a0af8a9e4435e30b857e789b4172ec47"} Aug 04 18:17:35.078189 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.078130 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:35.183931 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.183895 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs"] Aug 04 18:17:35.190909 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.190785 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" Aug 04 18:17:35.193251 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.193224 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Aug 04 18:17:35.194105 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.194088 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-599sw\"" Aug 04 18:17:35.202419 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.202392 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr"] Aug 04 18:17:35.206573 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.206381 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7d8m\" (UniqueName: \"kubernetes.io/projected/69af5af2-c017-4455-b51d-03fe560a578b-kube-api-access-l7d8m\") pod \"kserve-controller-manager-d6f4b65b8-2cdvs\" (UID: \"69af5af2-c017-4455-b51d-03fe560a578b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" Aug 04 18:17:35.206573 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.206458 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69af5af2-c017-4455-b51d-03fe560a578b-cert\") pod \"kserve-controller-manager-d6f4b65b8-2cdvs\" (UID: \"69af5af2-c017-4455-b51d-03fe560a578b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" Aug 04 18:17:35.208330 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.208245 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs"] Aug 04 18:17:35.208833 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.208495 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" Aug 04 18:17:35.214892 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.214475 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Aug 04 18:17:35.214892 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.214596 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dmqj4\"" Aug 04 18:17:35.216514 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.216424 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr"] Aug 04 18:17:35.237929 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.237889 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:35.307442 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.307413 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69af5af2-c017-4455-b51d-03fe560a578b-cert\") pod \"kserve-controller-manager-d6f4b65b8-2cdvs\" (UID: \"69af5af2-c017-4455-b51d-03fe560a578b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" Aug 04 18:17:35.307591 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.307469 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1caa4745-97d7-40d8-b66b-ee647e71a995-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7j4tr\" (UID: \"1caa4745-97d7-40d8-b66b-ee647e71a995\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" Aug 04 18:17:35.307591 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.307514 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spv8p\" (UniqueName: \"kubernetes.io/projected/1caa4745-97d7-40d8-b66b-ee647e71a995-kube-api-access-spv8p\") pod \"llmisvc-controller-manager-777c6bd7d5-7j4tr\" (UID: \"1caa4745-97d7-40d8-b66b-ee647e71a995\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" Aug 04 18:17:35.307591 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.307566 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7d8m\" (UniqueName: \"kubernetes.io/projected/69af5af2-c017-4455-b51d-03fe560a578b-kube-api-access-l7d8m\") pod \"kserve-controller-manager-d6f4b65b8-2cdvs\" (UID: \"69af5af2-c017-4455-b51d-03fe560a578b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" Aug 04 18:17:35.307591 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:17:35.307581 2526 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Aug 04 18:17:35.307805 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:17:35.307677 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69af5af2-c017-4455-b51d-03fe560a578b-cert podName:69af5af2-c017-4455-b51d-03fe560a578b nodeName:}" failed. No retries permitted until 2026-08-04 18:17:35.8076605 +0000 UTC m=+413.058943025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/69af5af2-c017-4455-b51d-03fe560a578b-cert") pod "kserve-controller-manager-d6f4b65b8-2cdvs" (UID: "69af5af2-c017-4455-b51d-03fe560a578b") : secret "kserve-webhook-server-cert" not found Aug 04 18:17:35.324389 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.324358 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7d8m\" (UniqueName: \"kubernetes.io/projected/69af5af2-c017-4455-b51d-03fe560a578b-kube-api-access-l7d8m\") pod \"kserve-controller-manager-d6f4b65b8-2cdvs\" (UID: \"69af5af2-c017-4455-b51d-03fe560a578b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" Aug 04 18:17:35.347774 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.347698 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:35.408365 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.408338 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1caa4745-97d7-40d8-b66b-ee647e71a995-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7j4tr\" (UID: \"1caa4745-97d7-40d8-b66b-ee647e71a995\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" Aug 04 18:17:35.408477 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.408378 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spv8p\" (UniqueName: \"kubernetes.io/projected/1caa4745-97d7-40d8-b66b-ee647e71a995-kube-api-access-spv8p\") pod \"llmisvc-controller-manager-777c6bd7d5-7j4tr\" (UID: \"1caa4745-97d7-40d8-b66b-ee647e71a995\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" Aug 04 18:17:35.411037 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.410982 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1caa4745-97d7-40d8-b66b-ee647e71a995-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7j4tr\" (UID: \"1caa4745-97d7-40d8-b66b-ee647e71a995\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" Aug 04 18:17:35.420705 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.420662 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spv8p\" (UniqueName: \"kubernetes.io/projected/1caa4745-97d7-40d8-b66b-ee647e71a995-kube-api-access-spv8p\") pod \"llmisvc-controller-manager-777c6bd7d5-7j4tr\" (UID: \"1caa4745-97d7-40d8-b66b-ee647e71a995\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" Aug 04 18:17:35.468618 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.468528 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:35.527689 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.527666 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" Aug 04 18:17:35.590271 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.590231 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:35.646082 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.646038 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:35.717496 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.717473 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr"] Aug 04 18:17:35.795474 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.795440 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:35.812739 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.812706 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69af5af2-c017-4455-b51d-03fe560a578b-cert\") pod \"kserve-controller-manager-d6f4b65b8-2cdvs\" (UID: \"69af5af2-c017-4455-b51d-03fe560a578b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" Aug 04 18:17:35.815228 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:35.815205 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69af5af2-c017-4455-b51d-03fe560a578b-cert\") pod \"kserve-controller-manager-d6f4b65b8-2cdvs\" (UID: \"69af5af2-c017-4455-b51d-03fe560a578b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" Aug 04 18:17:36.110131 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:36.110095 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" Aug 04 18:17:36.571156 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:17:36.571123 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1caa4745_97d7_40d8_b66b_ee647e71a995.slice/crio-004d7efdcd4a38388642d4297945f506e0c98274b1ae218f5b07a4d12ce714bf WatchSource:0}: Error finding container 004d7efdcd4a38388642d4297945f506e0c98274b1ae218f5b07a4d12ce714bf: Status 404 returned error can't find the container with id 004d7efdcd4a38388642d4297945f506e0c98274b1ae218f5b07a4d12ce714bf Aug 04 18:17:36.636930 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:36.636892 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:17:36.637154 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:36.636977 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:17:36.637154 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:36.637033 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:17:36.667325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:36.667263 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" event={"ID":"1caa4745-97d7-40d8-b66b-ee647e71a995","Type":"ContainerStarted","Data":"004d7efdcd4a38388642d4297945f506e0c98274b1ae218f5b07a4d12ce714bf"} Aug 04 18:17:36.710150 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:36.710125 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs"] Aug 04 18:17:36.712417 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:17:36.712390 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69af5af2_c017_4455_b51d_03fe560a578b.slice/crio-3c2936efc713e6c46c3a96fbd7cb693c09fb20cf763e3d3f3c21071e7dd64377 WatchSource:0}: Error finding container 3c2936efc713e6c46c3a96fbd7cb693c09fb20cf763e3d3f3c21071e7dd64377: Status 404 returned error can't find the container with id 3c2936efc713e6c46c3a96fbd7cb693c09fb20cf763e3d3f3c21071e7dd64377 Aug 04 18:17:37.375394 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.375367 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5"] Aug 04 18:17:37.375830 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.375563 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" podUID="781daeeb-2bd8-4741-b4c4-c7222bc4a7d8" containerName="discovery" containerID="cri-o://e1c416a5b90af0b28ce92d36c1ef59abbe2c30ae70ac9af0068954f2da76b129" gracePeriod=30 Aug 04 18:17:37.692519 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.690283 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" event={"ID":"69af5af2-c017-4455-b51d-03fe560a578b","Type":"ContainerStarted","Data":"3c2936efc713e6c46c3a96fbd7cb693c09fb20cf763e3d3f3c21071e7dd64377"} Aug 04 18:17:37.698191 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.698160 2526 generic.go:356] "Generic (PLEG): container finished" podID="781daeeb-2bd8-4741-b4c4-c7222bc4a7d8" containerID="e1c416a5b90af0b28ce92d36c1ef59abbe2c30ae70ac9af0068954f2da76b129" exitCode=0 Aug 04 18:17:37.698360 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.698230 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" event={"ID":"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8","Type":"ContainerDied","Data":"e1c416a5b90af0b28ce92d36c1ef59abbe2c30ae70ac9af0068954f2da76b129"} Aug 04 18:17:37.700835 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.700806 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" event={"ID":"8f3e6bd2-9d7a-4354-8775-1a0f0292891c","Type":"ContainerStarted","Data":"c64f6b0bc80778e345ab4f38947b654924147920a000b5bd3cda34c9d299d5eb"} Aug 04 18:17:37.737647 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.737587 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" podStartSLOduration=2.3975651940000002 podStartE2EDuration="5.737567691s" totalImagesPullingTime="3.340002497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:37.736829995 +0000 UTC m=+414.988112543" watchObservedRunningTime="2026-08-04 18:17:37.737567691 +0000 UTC m=+414.988850239" Aug 04 18:17:37.785055 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.783833 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:37.829271 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.829222 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-csr-ca-configmap\") pod \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " Aug 04 18:17:37.829271 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.829276 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-cacerts\") pod \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " Aug 04 18:17:37.829558 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.829312 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-token\") pod \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " Aug 04 18:17:37.829558 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.829361 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-local-certs\") pod \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " Aug 04 18:17:37.829558 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.829441 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-csr-dns-cert\") pod \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " Aug 04 18:17:37.829558 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.829469 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-kubeconfig\") pod \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " Aug 04 18:17:37.829558 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.829505 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nzpq\" (UniqueName: \"kubernetes.io/projected/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-kube-api-access-8nzpq\") pod \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\" (UID: \"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8\") " Aug 04 18:17:37.830639 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.830376 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "781daeeb-2bd8-4741-b4c4-c7222bc4a7d8" (UID: "781daeeb-2bd8-4741-b4c4-c7222bc4a7d8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:17:37.836194 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.836155 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "781daeeb-2bd8-4741-b4c4-c7222bc4a7d8" (UID: "781daeeb-2bd8-4741-b4c4-c7222bc4a7d8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:17:37.836687 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.836644 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "781daeeb-2bd8-4741-b4c4-c7222bc4a7d8" (UID: "781daeeb-2bd8-4741-b4c4-c7222bc4a7d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:17:37.837023 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.836986 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-cacerts" (OuterVolumeSpecName: "cacerts") pod "781daeeb-2bd8-4741-b4c4-c7222bc4a7d8" (UID: "781daeeb-2bd8-4741-b4c4-c7222bc4a7d8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:17:37.838208 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.838004 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "781daeeb-2bd8-4741-b4c4-c7222bc4a7d8" (UID: "781daeeb-2bd8-4741-b4c4-c7222bc4a7d8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:17:37.838438 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.838335 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-kube-api-access-8nzpq" (OuterVolumeSpecName: "kube-api-access-8nzpq") pod "781daeeb-2bd8-4741-b4c4-c7222bc4a7d8" (UID: "781daeeb-2bd8-4741-b4c4-c7222bc4a7d8"). InnerVolumeSpecName "kube-api-access-8nzpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:17:37.838438 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.838408 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-local-certs" (OuterVolumeSpecName: "local-certs") pod "781daeeb-2bd8-4741-b4c4-c7222bc4a7d8" (UID: "781daeeb-2bd8-4741-b4c4-c7222bc4a7d8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:17:37.931064 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.930979 2526 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-csr-dns-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:17:37.931064 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.931018 2526 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-kubeconfig\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:17:37.931064 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.931034 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nzpq\" (UniqueName: \"kubernetes.io/projected/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-kube-api-access-8nzpq\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:17:37.931064 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.931050 2526 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-csr-ca-configmap\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:17:37.931064 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.931066 2526 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-cacerts\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:17:37.931421 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.931080 2526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-istio-token\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:17:37.931421 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:37.931094 2526 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8-local-certs\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:17:38.095658 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:38.095626 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:38.101100 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:38.101077 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:38.707130 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:38.707102 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" Aug 04 18:17:38.707579 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:38.707141 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5" event={"ID":"781daeeb-2bd8-4741-b4c4-c7222bc4a7d8","Type":"ContainerDied","Data":"867efab6a7f0c578cd6c4a05bdaeca570f769adafd3234bea52c158c6fcddfa0"} Aug 04 18:17:38.707579 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:38.707197 2526 scope.go:117] "RemoveContainer" containerID="e1c416a5b90af0b28ce92d36c1ef59abbe2c30ae70ac9af0068954f2da76b129" Aug 04 18:17:38.707579 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:38.707473 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:38.708934 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:38.708765 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv" Aug 04 18:17:38.769863 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:38.769580 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5"] Aug 04 18:17:38.776429 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:38.776390 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gndv5"] Aug 04 18:17:39.182084 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:39.181999 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="781daeeb-2bd8-4741-b4c4-c7222bc4a7d8" path="/var/lib/kubelet/pods/781daeeb-2bd8-4741-b4c4-c7222bc4a7d8/volumes" Aug 04 18:17:39.633678 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:39.633652 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" Aug 04 18:17:39.715849 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:39.715812 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" event={"ID":"69af5af2-c017-4455-b51d-03fe560a578b","Type":"ContainerStarted","Data":"406d9dac6646b2439a5fd02b4e9f9a45d8d6641acaab7e19418230fba034d49c"} Aug 04 18:17:39.770980 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:39.770902 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" podStartSLOduration=2.188403778 podStartE2EDuration="4.770880483s" totalImagesPullingTime="2.582476705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:39.76851186 +0000 UTC m=+417.019794408" watchObservedRunningTime="2026-08-04 18:17:39.770880483 +0000 UTC m=+417.022163030" Aug 04 18:17:40.721785 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:40.721750 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" event={"ID":"1caa4745-97d7-40d8-b66b-ee647e71a995","Type":"ContainerStarted","Data":"28e9c8f6e45826f63499e4668a8160e80847c652c585413829378cd1a9ded3c7"} Aug 04 18:17:40.722254 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:40.721829 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" Aug 04 18:17:40.722254 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:40.722217 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" Aug 04 18:17:40.749713 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:40.749671 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" podStartSLOduration=2.231477046 podStartE2EDuration="5.749653274s" totalImagesPullingTime="3.518176228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:40.747951042 +0000 UTC m=+417.999233588" watchObservedRunningTime="2026-08-04 18:17:40.749653274 +0000 UTC m=+418.000935821" Aug 04 18:17:42.649247 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:17:42.649223 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-gkwpb" Aug 04 18:18:11.728538 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:11.728456 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" Aug 04 18:18:11.731267 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:11.731245 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" Aug 04 18:18:23.901796 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:23.901764 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:24.009854 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:24.009817 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:39.553488 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:39.553451 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:40.922253 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:40.922215 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:41.772839 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:41.772807 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg"] Aug 04 18:18:41.773126 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:41.773114 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781daeeb-2bd8-4741-b4c4-c7222bc4a7d8" containerName="discovery" Aug 04 18:18:41.773178 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:41.773128 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="781daeeb-2bd8-4741-b4c4-c7222bc4a7d8" containerName="discovery" Aug 04 18:18:41.773215 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:41.773193 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="781daeeb-2bd8-4741-b4c4-c7222bc4a7d8" containerName="discovery" Aug 04 18:18:41.774841 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:41.774825 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg" Aug 04 18:18:41.777404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:41.777383 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Aug 04 18:18:41.777529 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:41.777390 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 18:18:41.778316 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:41.778292 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 18:18:41.778316 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:41.778292 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-h9xz8\"" Aug 04 18:18:41.788606 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:41.788580 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg"] Aug 04 18:18:41.927738 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:41.927704 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f5fp\" (UniqueName: \"kubernetes.io/projected/e92e0caa-ebcb-4775-92c1-f66c63d895e0-kube-api-access-9f5fp\") pod \"dns-operator-controller-manager-5985564776-hl5wg\" (UID: \"e92e0caa-ebcb-4775-92c1-f66c63d895e0\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg" Aug 04 18:18:42.028579 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:42.028493 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f5fp\" (UniqueName: \"kubernetes.io/projected/e92e0caa-ebcb-4775-92c1-f66c63d895e0-kube-api-access-9f5fp\") pod \"dns-operator-controller-manager-5985564776-hl5wg\" (UID: \"e92e0caa-ebcb-4775-92c1-f66c63d895e0\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg" Aug 04 18:18:42.043076 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:42.043046 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f5fp\" (UniqueName: \"kubernetes.io/projected/e92e0caa-ebcb-4775-92c1-f66c63d895e0-kube-api-access-9f5fp\") pod \"dns-operator-controller-manager-5985564776-hl5wg\" (UID: \"e92e0caa-ebcb-4775-92c1-f66c63d895e0\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg" Aug 04 18:18:42.084784 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:42.084762 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg" Aug 04 18:18:42.215051 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:42.215027 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg"] Aug 04 18:18:42.217880 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:18:42.217856 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92e0caa_ebcb_4775_92c1_f66c63d895e0.slice/crio-bc714bd852c9f078a8d75ea3c3122dfd8f6de12f79c63ff403b7e08cfba3f7ad WatchSource:0}: Error finding container bc714bd852c9f078a8d75ea3c3122dfd8f6de12f79c63ff403b7e08cfba3f7ad: Status 404 returned error can't find the container with id bc714bd852c9f078a8d75ea3c3122dfd8f6de12f79c63ff403b7e08cfba3f7ad Aug 04 18:18:42.304751 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:42.304677 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:42.760693 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:42.760667 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:42.965482 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:42.965448 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg" event={"ID":"e92e0caa-ebcb-4775-92c1-f66c63d895e0","Type":"ContainerStarted","Data":"bc714bd852c9f078a8d75ea3c3122dfd8f6de12f79c63ff403b7e08cfba3f7ad"} Aug 04 18:18:43.118332 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:43.118245 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:43.671181 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:43.671145 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:45.046125 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:45.046107 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Aug 04 18:18:45.504297 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:45.504257 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:45.981330 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:45.981298 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg" event={"ID":"e92e0caa-ebcb-4775-92c1-f66c63d895e0","Type":"ContainerStarted","Data":"e22e7a617dd23bc4582cf880e528e14d7c452b7cd5eb9ab82274804074433832"} Aug 04 18:18:45.981481 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:45.981441 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg" Aug 04 18:18:46.498607 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.498546 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg" podStartSLOduration=2.674949167 podStartE2EDuration="5.498525303s" totalImagesPullingTime="2.823576136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:18:46.039723089 +0000 UTC m=+483.291005646" watchObservedRunningTime="2026-08-04 18:18:46.498525303 +0000 UTC m=+483.749807849" Aug 04 18:18:46.499421 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.499400 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk"] Aug 04 18:18:46.501805 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.501787 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" Aug 04 18:18:46.507128 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.507109 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rrnp5\"" Aug 04 18:18:46.526471 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.526450 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk"] Aug 04 18:18:46.551153 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.551120 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:46.562152 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.562131 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/466610b9-c8c4-4c71-9f4e-cec7ecc3b96e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-jfgpk\" (UID: \"466610b9-c8c4-4c71-9f4e-cec7ecc3b96e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" Aug 04 18:18:46.562255 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.562190 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7vql\" (UniqueName: \"kubernetes.io/projected/466610b9-c8c4-4c71-9f4e-cec7ecc3b96e-kube-api-access-r7vql\") pod \"kuadrant-operator-controller-manager-58fc56757c-jfgpk\" (UID: \"466610b9-c8c4-4c71-9f4e-cec7ecc3b96e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" Aug 04 18:18:46.663218 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.663187 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7vql\" (UniqueName: \"kubernetes.io/projected/466610b9-c8c4-4c71-9f4e-cec7ecc3b96e-kube-api-access-r7vql\") pod \"kuadrant-operator-controller-manager-58fc56757c-jfgpk\" (UID: \"466610b9-c8c4-4c71-9f4e-cec7ecc3b96e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" Aug 04 18:18:46.663361 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.663233 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/466610b9-c8c4-4c71-9f4e-cec7ecc3b96e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-jfgpk\" (UID: \"466610b9-c8c4-4c71-9f4e-cec7ecc3b96e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" Aug 04 18:18:46.663622 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.663602 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/466610b9-c8c4-4c71-9f4e-cec7ecc3b96e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-jfgpk\" (UID: \"466610b9-c8c4-4c71-9f4e-cec7ecc3b96e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" Aug 04 18:18:46.681259 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.681239 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7vql\" (UniqueName: \"kubernetes.io/projected/466610b9-c8c4-4c71-9f4e-cec7ecc3b96e-kube-api-access-r7vql\") pod \"kuadrant-operator-controller-manager-58fc56757c-jfgpk\" (UID: \"466610b9-c8c4-4c71-9f4e-cec7ecc3b96e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" Aug 04 18:18:46.795206 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.795139 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:46.811515 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.811484 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" Aug 04 18:18:46.906511 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.906460 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:46.961593 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.961566 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk"] Aug 04 18:18:46.963888 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:18:46.963857 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466610b9_c8c4_4c71_9f4e_cec7ecc3b96e.slice/crio-e822e2b07fe0db18bb4159035b1b7ccbca945719abbe02465e7792abf23df16e WatchSource:0}: Error finding container e822e2b07fe0db18bb4159035b1b7ccbca945719abbe02465e7792abf23df16e: Status 404 returned error can't find the container with id e822e2b07fe0db18bb4159035b1b7ccbca945719abbe02465e7792abf23df16e Aug 04 18:18:46.987230 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:46.987205 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" event={"ID":"466610b9-c8c4-4c71-9f4e-cec7ecc3b96e","Type":"ContainerStarted","Data":"e822e2b07fe0db18bb4159035b1b7ccbca945719abbe02465e7792abf23df16e"} Aug 04 18:18:47.034258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:47.034224 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:49.185760 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:49.185734 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-dmvk7"] Aug 04 18:18:49.187880 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:49.187862 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" Aug 04 18:18:49.196606 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:49.196572 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6dw2s\"" Aug 04 18:18:49.214630 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:49.214606 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-dmvk7"] Aug 04 18:18:49.283347 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:49.283311 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d26q\" (UniqueName: \"kubernetes.io/projected/23b21430-239b-4799-8119-ed05095d4ee9-kube-api-access-2d26q\") pod \"authorino-operator-68d88948d8-dmvk7\" (UID: \"23b21430-239b-4799-8119-ed05095d4ee9\") " pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" Aug 04 18:18:49.384799 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:49.384767 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d26q\" (UniqueName: \"kubernetes.io/projected/23b21430-239b-4799-8119-ed05095d4ee9-kube-api-access-2d26q\") pod \"authorino-operator-68d88948d8-dmvk7\" (UID: \"23b21430-239b-4799-8119-ed05095d4ee9\") " pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" Aug 04 18:18:49.397738 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:49.397710 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d26q\" (UniqueName: \"kubernetes.io/projected/23b21430-239b-4799-8119-ed05095d4ee9-kube-api-access-2d26q\") pod \"authorino-operator-68d88948d8-dmvk7\" (UID: \"23b21430-239b-4799-8119-ed05095d4ee9\") " pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" Aug 04 18:18:49.500773 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:49.500741 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" Aug 04 18:18:49.557122 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:49.557074 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:49.659246 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:49.659218 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-dmvk7"] Aug 04 18:18:49.661415 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:18:49.661386 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b21430_239b_4799_8119_ed05095d4ee9.slice/crio-812d686413d7210f4462c3ff4b10e7af81532c2df9613a1b4bcb0d9a78a256a8 WatchSource:0}: Error finding container 812d686413d7210f4462c3ff4b10e7af81532c2df9613a1b4bcb0d9a78a256a8: Status 404 returned error can't find the container with id 812d686413d7210f4462c3ff4b10e7af81532c2df9613a1b4bcb0d9a78a256a8 Aug 04 18:18:50.001403 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:50.001370 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" event={"ID":"23b21430-239b-4799-8119-ed05095d4ee9","Type":"ContainerStarted","Data":"812d686413d7210f4462c3ff4b10e7af81532c2df9613a1b4bcb0d9a78a256a8"} Aug 04 18:18:51.037859 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:51.037826 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr"] Aug 04 18:18:51.040236 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:51.040202 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:51.040560 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:51.040534 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" Aug 04 18:18:51.044144 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:51.044126 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mjhpc\"" Aug 04 18:18:51.058329 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:51.058308 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr"] Aug 04 18:18:51.101098 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:51.101067 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr5dc\" (UniqueName: \"kubernetes.io/projected/07e3358e-5ef7-4eec-b4fe-ba1679387251-kube-api-access-nr5dc\") pod \"limitador-operator-controller-manager-6c549b6d85-vmxbr\" (UID: \"07e3358e-5ef7-4eec-b4fe-ba1679387251\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" Aug 04 18:18:51.201818 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:51.201706 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr5dc\" (UniqueName: \"kubernetes.io/projected/07e3358e-5ef7-4eec-b4fe-ba1679387251-kube-api-access-nr5dc\") pod \"limitador-operator-controller-manager-6c549b6d85-vmxbr\" (UID: \"07e3358e-5ef7-4eec-b4fe-ba1679387251\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" Aug 04 18:18:51.214658 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:51.214634 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr5dc\" (UniqueName: \"kubernetes.io/projected/07e3358e-5ef7-4eec-b4fe-ba1679387251-kube-api-access-nr5dc\") pod \"limitador-operator-controller-manager-6c549b6d85-vmxbr\" (UID: \"07e3358e-5ef7-4eec-b4fe-ba1679387251\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" Aug 04 18:18:51.353858 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:51.353769 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" Aug 04 18:18:51.531712 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:51.531689 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr"] Aug 04 18:18:51.533948 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:18:51.533872 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e3358e_5ef7_4eec_b4fe_ba1679387251.slice/crio-6dec9e5df23189c483a88f75facdfc7ef399175537342a10f2cf47cc1d342606 WatchSource:0}: Error finding container 6dec9e5df23189c483a88f75facdfc7ef399175537342a10f2cf47cc1d342606: Status 404 returned error can't find the container with id 6dec9e5df23189c483a88f75facdfc7ef399175537342a10f2cf47cc1d342606 Aug 04 18:18:52.011746 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:52.011703 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" event={"ID":"23b21430-239b-4799-8119-ed05095d4ee9","Type":"ContainerStarted","Data":"9d3309a444a28ae086a7a025a53cb7153c28be82bc26498dbe79c62309be5c77"} Aug 04 18:18:52.011890 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:52.011780 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" Aug 04 18:18:52.013401 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:52.013366 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" event={"ID":"466610b9-c8c4-4c71-9f4e-cec7ecc3b96e","Type":"ContainerStarted","Data":"83b161302ca1f6217949908b550ec61386b82ec2915b6b18224d9248cee9f24a"} Aug 04 18:18:52.013518 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:52.013471 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" Aug 04 18:18:52.014556 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:52.014532 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" event={"ID":"07e3358e-5ef7-4eec-b4fe-ba1679387251","Type":"ContainerStarted","Data":"6dec9e5df23189c483a88f75facdfc7ef399175537342a10f2cf47cc1d342606"} Aug 04 18:18:52.036019 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:52.035977 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" podStartSLOduration=0.799267424 podStartE2EDuration="3.035949642s" totalImagesPullingTime="2.236682218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:18:52.034820218 +0000 UTC m=+489.286102767" watchObservedRunningTime="2026-08-04 18:18:52.035949642 +0000 UTC m=+489.287232190" Aug 04 18:18:52.056046 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:52.056001 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" podStartSLOduration=1.820369934 podStartE2EDuration="6.05598519s" totalImagesPullingTime="4.235615256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:18:52.053896919 +0000 UTC m=+489.305179467" watchObservedRunningTime="2026-08-04 18:18:52.05598519 +0000 UTC m=+489.307267737" Aug 04 18:18:52.529072 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:52.529029 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:52.655163 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:52.655131 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:54.024896 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:54.024863 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" event={"ID":"07e3358e-5ef7-4eec-b4fe-ba1679387251","Type":"ContainerStarted","Data":"b0b21ddc393bc873cfae4896db1a5b467cc7d3ca5ac03cf69fc01b5cc9bcbb1e"} Aug 04 18:18:54.025377 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:54.025008 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" Aug 04 18:18:54.044063 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:54.044021 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" podStartSLOduration=2.408185683 podStartE2EDuration="4.044006765s" totalImagesPullingTime="1.635821082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:18:54.043983924 +0000 UTC m=+491.295266468" watchObservedRunningTime="2026-08-04 18:18:54.044006765 +0000 UTC m=+491.295289313" Aug 04 18:18:54.182573 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:54.182537 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:54.399768 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:54.399693 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:56.989639 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:56.989610 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg" Aug 04 18:18:57.104790 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:57.104754 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:57.217718 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:57.217687 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:58.093469 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:58.093435 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:58.193623 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:58.193587 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:58.971643 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:58.971604 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:59.115786 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:18:59.115757 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:03.021319 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:03.021290 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" Aug 04 18:19:03.021673 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:03.021363 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" Aug 04 18:19:05.032486 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.032453 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" Aug 04 18:19:05.041830 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.041808 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk"] Aug 04 18:19:05.041989 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.041952 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" podUID="466610b9-c8c4-4c71-9f4e-cec7ecc3b96e" containerName="manager" containerID="cri-o://83b161302ca1f6217949908b550ec61386b82ec2915b6b18224d9248cee9f24a" gracePeriod=2 Aug 04 18:19:05.067246 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.067224 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk"] Aug 04 18:19:05.083743 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.083658 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr"] Aug 04 18:19:05.083889 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.083872 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" podUID="07e3358e-5ef7-4eec-b4fe-ba1679387251" containerName="manager" containerID="cri-o://b0b21ddc393bc873cfae4896db1a5b467cc7d3ca5ac03cf69fc01b5cc9bcbb1e" gracePeriod=2 Aug 04 18:19:05.085355 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.085331 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf"] Aug 04 18:19:05.085646 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.085631 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="466610b9-c8c4-4c71-9f4e-cec7ecc3b96e" containerName="manager" Aug 04 18:19:05.085721 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.085649 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="466610b9-c8c4-4c71-9f4e-cec7ecc3b96e" containerName="manager" Aug 04 18:19:05.085721 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.085712 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="466610b9-c8c4-4c71-9f4e-cec7ecc3b96e" containerName="manager" Aug 04 18:19:05.088640 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.088621 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" Aug 04 18:19:05.092304 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.092275 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr"] Aug 04 18:19:05.100557 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:19:05.100498 2526 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"07e3358e-5ef7-4eec-b4fe-ba1679387251\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-6c549b6d85-vmxbr\": pods \"limitador-operator-controller-manager-6c549b6d85-vmxbr\" not found" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" Aug 04 18:19:05.103132 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.103107 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf"] Aug 04 18:19:05.105144 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:19:05.104906 2526 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-jfgpk\" is forbidden: User \"system:node:ip-10-0-136-165.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-165.ec2.internal' and this object" podUID="466610b9-c8c4-4c71-9f4e-cec7ecc3b96e" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" Aug 04 18:19:05.106898 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.106868 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk"] Aug 04 18:19:05.107334 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.107313 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e3358e-5ef7-4eec-b4fe-ba1679387251" containerName="manager" Aug 04 18:19:05.107334 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.107337 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e3358e-5ef7-4eec-b4fe-ba1679387251" containerName="manager" Aug 04 18:19:05.107529 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.107445 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e3358e-5ef7-4eec-b4fe-ba1679387251" containerName="manager" Aug 04 18:19:05.107785 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:19:05.107762 2526 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-vmxbr\" is forbidden: User \"system:node:ip-10-0-136-165.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-165.ec2.internal' and this object" podUID="07e3358e-5ef7-4eec-b4fe-ba1679387251" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" Aug 04 18:19:05.109811 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:19:05.109784 2526 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-jfgpk\" is forbidden: User \"system:node:ip-10-0-136-165.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-165.ec2.internal' and this object" podUID="466610b9-c8c4-4c71-9f4e-cec7ecc3b96e" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" Aug 04 18:19:05.110344 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.110330 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk" Aug 04 18:19:05.124684 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.124656 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk"] Aug 04 18:19:05.148535 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:19:05.148504 2526 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-jfgpk\" is forbidden: User \"system:node:ip-10-0-136-165.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-165.ec2.internal' and this object" podUID="466610b9-c8c4-4c71-9f4e-cec7ecc3b96e" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" Aug 04 18:19:05.173319 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:19:05.173290 2526 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-vmxbr\" is forbidden: User \"system:node:ip-10-0-136-165.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-165.ec2.internal' and this object" podUID="07e3358e-5ef7-4eec-b4fe-ba1679387251" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" Aug 04 18:19:05.206639 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.206614 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61441698-1973-4202-bdee-58a256a4e92b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-qtqmf\" (UID: \"61441698-1973-4202-bdee-58a256a4e92b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" Aug 04 18:19:05.206763 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.206659 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcd7f\" (UniqueName: \"kubernetes.io/projected/2256ff0e-84f0-4c89-a5e8-6809a4be5154-kube-api-access-dcd7f\") pod \"limitador-operator-controller-manager-6c549b6d85-mtwwk\" (UID: \"2256ff0e-84f0-4c89-a5e8-6809a4be5154\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk" Aug 04 18:19:05.206763 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.206694 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phxtk\" (UniqueName: \"kubernetes.io/projected/61441698-1973-4202-bdee-58a256a4e92b-kube-api-access-phxtk\") pod \"kuadrant-operator-controller-manager-58fc56757c-qtqmf\" (UID: \"61441698-1973-4202-bdee-58a256a4e92b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" Aug 04 18:19:05.307651 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.307622 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61441698-1973-4202-bdee-58a256a4e92b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-qtqmf\" (UID: \"61441698-1973-4202-bdee-58a256a4e92b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" Aug 04 18:19:05.307750 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.307689 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd7f\" (UniqueName: \"kubernetes.io/projected/2256ff0e-84f0-4c89-a5e8-6809a4be5154-kube-api-access-dcd7f\") pod \"limitador-operator-controller-manager-6c549b6d85-mtwwk\" (UID: \"2256ff0e-84f0-4c89-a5e8-6809a4be5154\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk" Aug 04 18:19:05.307750 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.307716 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phxtk\" (UniqueName: \"kubernetes.io/projected/61441698-1973-4202-bdee-58a256a4e92b-kube-api-access-phxtk\") pod \"kuadrant-operator-controller-manager-58fc56757c-qtqmf\" (UID: \"61441698-1973-4202-bdee-58a256a4e92b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" Aug 04 18:19:05.308012 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.307989 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61441698-1973-4202-bdee-58a256a4e92b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-qtqmf\" (UID: \"61441698-1973-4202-bdee-58a256a4e92b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" Aug 04 18:19:05.317804 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.317781 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" Aug 04 18:19:05.320045 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.320026 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcd7f\" (UniqueName: \"kubernetes.io/projected/2256ff0e-84f0-4c89-a5e8-6809a4be5154-kube-api-access-dcd7f\") pod \"limitador-operator-controller-manager-6c549b6d85-mtwwk\" (UID: \"2256ff0e-84f0-4c89-a5e8-6809a4be5154\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk" Aug 04 18:19:05.320045 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.320037 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phxtk\" (UniqueName: \"kubernetes.io/projected/61441698-1973-4202-bdee-58a256a4e92b-kube-api-access-phxtk\") pod \"kuadrant-operator-controller-manager-58fc56757c-qtqmf\" (UID: \"61441698-1973-4202-bdee-58a256a4e92b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" Aug 04 18:19:05.320692 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.320677 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" Aug 04 18:19:05.408896 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.408876 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7vql\" (UniqueName: \"kubernetes.io/projected/466610b9-c8c4-4c71-9f4e-cec7ecc3b96e-kube-api-access-r7vql\") pod \"466610b9-c8c4-4c71-9f4e-cec7ecc3b96e\" (UID: \"466610b9-c8c4-4c71-9f4e-cec7ecc3b96e\") " Aug 04 18:19:05.409025 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.408912 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/466610b9-c8c4-4c71-9f4e-cec7ecc3b96e-extensions-socket-volume\") pod \"466610b9-c8c4-4c71-9f4e-cec7ecc3b96e\" (UID: \"466610b9-c8c4-4c71-9f4e-cec7ecc3b96e\") " Aug 04 18:19:05.409025 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.408935 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr5dc\" (UniqueName: \"kubernetes.io/projected/07e3358e-5ef7-4eec-b4fe-ba1679387251-kube-api-access-nr5dc\") pod \"07e3358e-5ef7-4eec-b4fe-ba1679387251\" (UID: \"07e3358e-5ef7-4eec-b4fe-ba1679387251\") " Aug 04 18:19:05.409347 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.409326 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466610b9-c8c4-4c71-9f4e-cec7ecc3b96e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "466610b9-c8c4-4c71-9f4e-cec7ecc3b96e" (UID: "466610b9-c8c4-4c71-9f4e-cec7ecc3b96e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:19:05.410924 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.410902 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466610b9-c8c4-4c71-9f4e-cec7ecc3b96e-kube-api-access-r7vql" (OuterVolumeSpecName: "kube-api-access-r7vql") pod "466610b9-c8c4-4c71-9f4e-cec7ecc3b96e" (UID: "466610b9-c8c4-4c71-9f4e-cec7ecc3b96e"). InnerVolumeSpecName "kube-api-access-r7vql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:19:05.410924 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.410913 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e3358e-5ef7-4eec-b4fe-ba1679387251-kube-api-access-nr5dc" (OuterVolumeSpecName: "kube-api-access-nr5dc") pod "07e3358e-5ef7-4eec-b4fe-ba1679387251" (UID: "07e3358e-5ef7-4eec-b4fe-ba1679387251"). InnerVolumeSpecName "kube-api-access-nr5dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:19:05.485375 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.485356 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" Aug 04 18:19:05.491241 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.491223 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk" Aug 04 18:19:05.510067 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.510037 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7vql\" (UniqueName: \"kubernetes.io/projected/466610b9-c8c4-4c71-9f4e-cec7ecc3b96e-kube-api-access-r7vql\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:19:05.510152 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.510072 2526 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/466610b9-c8c4-4c71-9f4e-cec7ecc3b96e-extensions-socket-volume\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:19:05.510152 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.510087 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr5dc\" (UniqueName: \"kubernetes.io/projected/07e3358e-5ef7-4eec-b4fe-ba1679387251-kube-api-access-nr5dc\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:19:05.632005 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.631956 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf"] Aug 04 18:19:05.639039 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:19:05.638954 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61441698_1973_4202_bdee_58a256a4e92b.slice/crio-90306ee23051499def13f56ee4b1632d0c3adab4e665291f1d75e1be44b8c36d WatchSource:0}: Error finding container 90306ee23051499def13f56ee4b1632d0c3adab4e665291f1d75e1be44b8c36d: Status 404 returned error can't find the container with id 90306ee23051499def13f56ee4b1632d0c3adab4e665291f1d75e1be44b8c36d Aug 04 18:19:05.650878 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:05.650856 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk"] Aug 04 18:19:05.654256 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:19:05.654236 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2256ff0e_84f0_4c89_a5e8_6809a4be5154.slice/crio-b3b3791bb4df643ccfe948836411eaf8bff59e83fc6411030d9310c962423fe5 WatchSource:0}: Error finding container b3b3791bb4df643ccfe948836411eaf8bff59e83fc6411030d9310c962423fe5: Status 404 returned error can't find the container with id b3b3791bb4df643ccfe948836411eaf8bff59e83fc6411030d9310c962423fe5 Aug 04 18:19:06.012853 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.012801 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:06.087917 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.087886 2526 generic.go:356] "Generic (PLEG): container finished" podID="07e3358e-5ef7-4eec-b4fe-ba1679387251" containerID="b0b21ddc393bc873cfae4896db1a5b467cc7d3ca5ac03cf69fc01b5cc9bcbb1e" exitCode=0 Aug 04 18:19:06.088328 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.087944 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-vmxbr" Aug 04 18:19:06.088328 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.088005 2526 scope.go:117] "RemoveContainer" containerID="b0b21ddc393bc873cfae4896db1a5b467cc7d3ca5ac03cf69fc01b5cc9bcbb1e" Aug 04 18:19:06.089798 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.089775 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk" event={"ID":"2256ff0e-84f0-4c89-a5e8-6809a4be5154","Type":"ContainerStarted","Data":"693faa6b7322064e441550c868f6286feb057466e9aa7d24d2cdd2eb109d9d10"} Aug 04 18:19:06.089890 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.089825 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk" event={"ID":"2256ff0e-84f0-4c89-a5e8-6809a4be5154","Type":"ContainerStarted","Data":"b3b3791bb4df643ccfe948836411eaf8bff59e83fc6411030d9310c962423fe5"} Aug 04 18:19:06.089890 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.089876 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk" Aug 04 18:19:06.091636 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.091615 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" event={"ID":"61441698-1973-4202-bdee-58a256a4e92b","Type":"ContainerStarted","Data":"30d42fcb1ea6da5eb1ad8bd33b1a4254b1fac47128cab1fc56f8643ff16e3daa"} Aug 04 18:19:06.091732 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.091651 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" event={"ID":"61441698-1973-4202-bdee-58a256a4e92b","Type":"ContainerStarted","Data":"90306ee23051499def13f56ee4b1632d0c3adab4e665291f1d75e1be44b8c36d"} Aug 04 18:19:06.091732 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.091676 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" Aug 04 18:19:06.093114 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.093087 2526 generic.go:356] "Generic (PLEG): container finished" podID="466610b9-c8c4-4c71-9f4e-cec7ecc3b96e" containerID="83b161302ca1f6217949908b550ec61386b82ec2915b6b18224d9248cee9f24a" exitCode=0 Aug 04 18:19:06.093198 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.093134 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jfgpk" Aug 04 18:19:06.096697 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.096680 2526 scope.go:117] "RemoveContainer" containerID="b0b21ddc393bc873cfae4896db1a5b467cc7d3ca5ac03cf69fc01b5cc9bcbb1e" Aug 04 18:19:06.096948 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:19:06.096930 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b21ddc393bc873cfae4896db1a5b467cc7d3ca5ac03cf69fc01b5cc9bcbb1e\": container with ID starting with b0b21ddc393bc873cfae4896db1a5b467cc7d3ca5ac03cf69fc01b5cc9bcbb1e not found: ID does not exist" containerID="b0b21ddc393bc873cfae4896db1a5b467cc7d3ca5ac03cf69fc01b5cc9bcbb1e" Aug 04 18:19:06.097042 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.096958 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b21ddc393bc873cfae4896db1a5b467cc7d3ca5ac03cf69fc01b5cc9bcbb1e"} err="failed to get container status \"b0b21ddc393bc873cfae4896db1a5b467cc7d3ca5ac03cf69fc01b5cc9bcbb1e\": rpc error: code = NotFound desc = could not find container \"b0b21ddc393bc873cfae4896db1a5b467cc7d3ca5ac03cf69fc01b5cc9bcbb1e\": container with ID starting with b0b21ddc393bc873cfae4896db1a5b467cc7d3ca5ac03cf69fc01b5cc9bcbb1e not found: ID does not exist" Aug 04 18:19:06.097042 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.096993 2526 scope.go:117] "RemoveContainer" containerID="83b161302ca1f6217949908b550ec61386b82ec2915b6b18224d9248cee9f24a" Aug 04 18:19:06.107111 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.107095 2526 scope.go:117] "RemoveContainer" containerID="83b161302ca1f6217949908b550ec61386b82ec2915b6b18224d9248cee9f24a" Aug 04 18:19:06.107375 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:19:06.107357 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b161302ca1f6217949908b550ec61386b82ec2915b6b18224d9248cee9f24a\": container with ID starting with 83b161302ca1f6217949908b550ec61386b82ec2915b6b18224d9248cee9f24a not found: ID does not exist" containerID="83b161302ca1f6217949908b550ec61386b82ec2915b6b18224d9248cee9f24a" Aug 04 18:19:06.107433 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.107380 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b161302ca1f6217949908b550ec61386b82ec2915b6b18224d9248cee9f24a"} err="failed to get container status \"83b161302ca1f6217949908b550ec61386b82ec2915b6b18224d9248cee9f24a\": rpc error: code = NotFound desc = could not find container \"83b161302ca1f6217949908b550ec61386b82ec2915b6b18224d9248cee9f24a\": container with ID starting with 83b161302ca1f6217949908b550ec61386b82ec2915b6b18224d9248cee9f24a not found: ID does not exist" Aug 04 18:19:06.135222 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.135184 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk" podStartSLOduration=1.135172651 podStartE2EDuration="1.135172651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:19:06.133112644 +0000 UTC m=+503.384395192" watchObservedRunningTime="2026-08-04 18:19:06.135172651 +0000 UTC m=+503.386455194" Aug 04 18:19:06.163841 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.163799 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" podStartSLOduration=1.163787167 podStartE2EDuration="1.163787167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:19:06.16286407 +0000 UTC m=+503.414146616" watchObservedRunningTime="2026-08-04 18:19:06.163787167 +0000 UTC m=+503.415069714" Aug 04 18:19:06.189042 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.189021 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:06.289532 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:06.289464 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:07.179984 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:07.179940 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07e3358e-5ef7-4eec-b4fe-ba1679387251" path="/var/lib/kubelet/pods/07e3358e-5ef7-4eec-b4fe-ba1679387251/volumes" Aug 04 18:19:07.180330 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:07.180292 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="466610b9-c8c4-4c71-9f4e-cec7ecc3b96e" path="/var/lib/kubelet/pods/466610b9-c8c4-4c71-9f4e-cec7ecc3b96e/volumes" Aug 04 18:19:08.937371 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:08.937331 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:09.036235 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:09.036202 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:11.585175 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:11.585141 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:11.684937 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:11.684897 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:16.937790 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:16.937757 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:17.062481 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:17.062448 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:17.102252 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:17.102233 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" Aug 04 18:19:17.102372 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:17.102294 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk" Aug 04 18:19:17.394523 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:17.394489 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:17.564087 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:17.564049 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:17.905303 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:17.905259 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:18.078828 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:18.078799 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:21.843306 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:21.843273 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf"] Aug 04 18:19:21.843700 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:21.843442 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" podUID="61441698-1973-4202-bdee-58a256a4e92b" containerName="manager" containerID="cri-o://30d42fcb1ea6da5eb1ad8bd33b1a4254b1fac47128cab1fc56f8643ff16e3daa" gracePeriod=10 Aug 04 18:19:22.095085 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.095030 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" Aug 04 18:19:22.176380 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.176348 2526 generic.go:356] "Generic (PLEG): container finished" podID="61441698-1973-4202-bdee-58a256a4e92b" containerID="30d42fcb1ea6da5eb1ad8bd33b1a4254b1fac47128cab1fc56f8643ff16e3daa" exitCode=0 Aug 04 18:19:22.176536 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.176416 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" event={"ID":"61441698-1973-4202-bdee-58a256a4e92b","Type":"ContainerDied","Data":"30d42fcb1ea6da5eb1ad8bd33b1a4254b1fac47128cab1fc56f8643ff16e3daa"} Aug 04 18:19:22.176536 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.176431 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" Aug 04 18:19:22.176536 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.176468 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf" event={"ID":"61441698-1973-4202-bdee-58a256a4e92b","Type":"ContainerDied","Data":"90306ee23051499def13f56ee4b1632d0c3adab4e665291f1d75e1be44b8c36d"} Aug 04 18:19:22.176536 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.176492 2526 scope.go:117] "RemoveContainer" containerID="30d42fcb1ea6da5eb1ad8bd33b1a4254b1fac47128cab1fc56f8643ff16e3daa" Aug 04 18:19:22.184414 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.184398 2526 scope.go:117] "RemoveContainer" containerID="30d42fcb1ea6da5eb1ad8bd33b1a4254b1fac47128cab1fc56f8643ff16e3daa" Aug 04 18:19:22.184673 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:19:22.184655 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d42fcb1ea6da5eb1ad8bd33b1a4254b1fac47128cab1fc56f8643ff16e3daa\": container with ID starting with 30d42fcb1ea6da5eb1ad8bd33b1a4254b1fac47128cab1fc56f8643ff16e3daa not found: ID does not exist" containerID="30d42fcb1ea6da5eb1ad8bd33b1a4254b1fac47128cab1fc56f8643ff16e3daa" Aug 04 18:19:22.184748 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.184684 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d42fcb1ea6da5eb1ad8bd33b1a4254b1fac47128cab1fc56f8643ff16e3daa"} err="failed to get container status \"30d42fcb1ea6da5eb1ad8bd33b1a4254b1fac47128cab1fc56f8643ff16e3daa\": rpc error: code = NotFound desc = could not find container \"30d42fcb1ea6da5eb1ad8bd33b1a4254b1fac47128cab1fc56f8643ff16e3daa\": container with ID starting with 30d42fcb1ea6da5eb1ad8bd33b1a4254b1fac47128cab1fc56f8643ff16e3daa not found: ID does not exist" Aug 04 18:19:22.237060 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.237037 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phxtk\" (UniqueName: \"kubernetes.io/projected/61441698-1973-4202-bdee-58a256a4e92b-kube-api-access-phxtk\") pod \"61441698-1973-4202-bdee-58a256a4e92b\" (UID: \"61441698-1973-4202-bdee-58a256a4e92b\") " Aug 04 18:19:22.237175 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.237137 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61441698-1973-4202-bdee-58a256a4e92b-extensions-socket-volume\") pod \"61441698-1973-4202-bdee-58a256a4e92b\" (UID: \"61441698-1973-4202-bdee-58a256a4e92b\") " Aug 04 18:19:22.237451 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.237430 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61441698-1973-4202-bdee-58a256a4e92b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "61441698-1973-4202-bdee-58a256a4e92b" (UID: "61441698-1973-4202-bdee-58a256a4e92b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:19:22.239208 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.239186 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61441698-1973-4202-bdee-58a256a4e92b-kube-api-access-phxtk" (OuterVolumeSpecName: "kube-api-access-phxtk") pod "61441698-1973-4202-bdee-58a256a4e92b" (UID: "61441698-1973-4202-bdee-58a256a4e92b"). InnerVolumeSpecName "kube-api-access-phxtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:19:22.337659 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.337634 2526 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61441698-1973-4202-bdee-58a256a4e92b-extensions-socket-volume\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:19:22.337659 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.337655 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phxtk\" (UniqueName: \"kubernetes.io/projected/61441698-1973-4202-bdee-58a256a4e92b-kube-api-access-phxtk\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:19:22.505803 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.505774 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf"] Aug 04 18:19:22.511758 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:22.511735 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-qtqmf"] Aug 04 18:19:23.181370 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:23.181339 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61441698-1973-4202-bdee-58a256a4e92b" path="/var/lib/kubelet/pods/61441698-1973-4202-bdee-58a256a4e92b/volumes" Aug 04 18:19:25.749951 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:25.749919 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:25.851151 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:25.851111 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:28.758101 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:28.758057 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:28.856398 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:28.856361 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:38.443384 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.443300 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf"] Aug 04 18:19:38.443846 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.443758 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61441698-1973-4202-bdee-58a256a4e92b" containerName="manager" Aug 04 18:19:38.443846 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.443775 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="61441698-1973-4202-bdee-58a256a4e92b" containerName="manager" Aug 04 18:19:38.443954 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.443899 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="61441698-1973-4202-bdee-58a256a4e92b" containerName="manager" Aug 04 18:19:38.446713 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.446692 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.450113 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.450093 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8qwdm\"" Aug 04 18:19:38.463778 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.463757 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf"] Aug 04 18:19:38.475451 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.475420 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:38.548398 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.548353 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:38.553335 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.553310 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.553439 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.553348 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.553439 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.553373 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.553439 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.553425 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.553573 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.553466 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.553573 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.553529 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.553573 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.553567 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.553665 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.553581 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrm7p\" (UniqueName: \"kubernetes.io/projected/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-kube-api-access-xrm7p\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.553665 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.553597 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.654832 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.654799 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.654993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.654845 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.654993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.654862 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrm7p\" (UniqueName: \"kubernetes.io/projected/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-kube-api-access-xrm7p\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.654993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.654879 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.654993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.654897 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.654993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.654919 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.654993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.654947 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.654993 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.654978 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.655352 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.655014 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.655411 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.655345 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.655463 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.655423 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.655531 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.655511 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.655584 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.655555 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.655720 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.655701 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.657289 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.657263 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.657386 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.657353 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.668110 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.668087 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.668364 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.668348 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrm7p\" (UniqueName: \"kubernetes.io/projected/39fbabce-fc4c-413a-ae9d-4672ba61e1d6-kube-api-access-xrm7p\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vpxf\" (UID: \"39fbabce-fc4c-413a-ae9d-4672ba61e1d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.758058 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.758034 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:38.891538 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.891514 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf"] Aug 04 18:19:38.893028 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:19:38.893001 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fbabce_fc4c_413a_ae9d_4672ba61e1d6.slice/crio-8c31445f4c1c92a1bcf16e2b66cc0b24e07d8b2b2f5f69c62ecf7f793ce253f2 WatchSource:0}: Error finding container 8c31445f4c1c92a1bcf16e2b66cc0b24e07d8b2b2f5f69c62ecf7f793ce253f2: Status 404 returned error can't find the container with id 8c31445f4c1c92a1bcf16e2b66cc0b24e07d8b2b2f5f69c62ecf7f793ce253f2 Aug 04 18:19:38.895097 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.895065 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:19:38.895152 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.895134 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:19:38.895192 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:38.895163 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:19:39.256759 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:39.256728 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" event={"ID":"39fbabce-fc4c-413a-ae9d-4672ba61e1d6","Type":"ContainerStarted","Data":"4603fbc6ddc64aa45aad9a959533f76cc361d11eff9b9c953d0a557b49132f9e"} Aug 04 18:19:39.256913 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:39.256774 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" event={"ID":"39fbabce-fc4c-413a-ae9d-4672ba61e1d6","Type":"ContainerStarted","Data":"8c31445f4c1c92a1bcf16e2b66cc0b24e07d8b2b2f5f69c62ecf7f793ce253f2"} Aug 04 18:19:39.279463 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:39.279426 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" podStartSLOduration=1.279412349 podStartE2EDuration="1.279412349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:19:39.278663789 +0000 UTC m=+536.529946340" watchObservedRunningTime="2026-08-04 18:19:39.279412349 +0000 UTC m=+536.530694897" Aug 04 18:19:39.758504 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:39.758479 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:39.762912 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:39.762891 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:40.260950 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.260918 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:40.261782 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.261764 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:19:40.386247 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.386195 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:40.390679 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.390656 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-xnqpf"] Aug 04 18:19:40.393832 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.393815 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" Aug 04 18:19:40.396292 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.396276 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vgpqr\"" Aug 04 18:19:40.396388 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.396301 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 04 18:19:40.402139 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.402119 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-xnqpf"] Aug 04 18:19:40.472503 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.472467 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldwvl\" (UniqueName: \"kubernetes.io/projected/103604c1-45a6-4eb5-bb86-71e0681f7197-kube-api-access-ldwvl\") pod \"limitador-limitador-859c4d96d6-xnqpf\" (UID: \"103604c1-45a6-4eb5-bb86-71e0681f7197\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" Aug 04 18:19:40.472642 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.472532 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/103604c1-45a6-4eb5-bb86-71e0681f7197-config-file\") pod \"limitador-limitador-859c4d96d6-xnqpf\" (UID: \"103604c1-45a6-4eb5-bb86-71e0681f7197\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" Aug 04 18:19:40.498050 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.498015 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-xnqpf"] Aug 04 18:19:40.510551 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.510525 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:40.573706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.573607 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/103604c1-45a6-4eb5-bb86-71e0681f7197-config-file\") pod \"limitador-limitador-859c4d96d6-xnqpf\" (UID: \"103604c1-45a6-4eb5-bb86-71e0681f7197\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" Aug 04 18:19:40.573706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.573702 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwvl\" (UniqueName: \"kubernetes.io/projected/103604c1-45a6-4eb5-bb86-71e0681f7197-kube-api-access-ldwvl\") pod \"limitador-limitador-859c4d96d6-xnqpf\" (UID: \"103604c1-45a6-4eb5-bb86-71e0681f7197\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" Aug 04 18:19:40.574329 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.574305 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/103604c1-45a6-4eb5-bb86-71e0681f7197-config-file\") pod \"limitador-limitador-859c4d96d6-xnqpf\" (UID: \"103604c1-45a6-4eb5-bb86-71e0681f7197\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" Aug 04 18:19:40.583300 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.583277 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwvl\" (UniqueName: \"kubernetes.io/projected/103604c1-45a6-4eb5-bb86-71e0681f7197-kube-api-access-ldwvl\") pod \"limitador-limitador-859c4d96d6-xnqpf\" (UID: \"103604c1-45a6-4eb5-bb86-71e0681f7197\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" Aug 04 18:19:40.584637 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.584602 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:40.705729 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.705691 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" Aug 04 18:19:40.841273 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:40.841211 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-xnqpf"] Aug 04 18:19:40.843990 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:19:40.843952 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103604c1_45a6_4eb5_bb86_71e0681f7197.slice/crio-966ddef9074cbc1da0ca02e19464b2074a29a35011815063142abfb008f80927 WatchSource:0}: Error finding container 966ddef9074cbc1da0ca02e19464b2074a29a35011815063142abfb008f80927: Status 404 returned error can't find the container with id 966ddef9074cbc1da0ca02e19464b2074a29a35011815063142abfb008f80927 Aug 04 18:19:41.266579 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.266546 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" event={"ID":"103604c1-45a6-4eb5-bb86-71e0681f7197","Type":"ContainerStarted","Data":"966ddef9074cbc1da0ca02e19464b2074a29a35011815063142abfb008f80927"} Aug 04 18:19:41.364281 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.364252 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-d5qfw"] Aug 04 18:19:41.368637 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.368618 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-d5qfw" Aug 04 18:19:41.373571 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.373550 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kn2sc\"" Aug 04 18:19:41.378067 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.378044 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-d5qfw"] Aug 04 18:19:41.412148 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.412120 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:41.480824 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.480801 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sr8c\" (UniqueName: \"kubernetes.io/projected/f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa-kube-api-access-5sr8c\") pod \"authorino-b54df94b4-d5qfw\" (UID: \"f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa\") " pod="kuadrant-system/authorino-b54df94b4-d5qfw" Aug 04 18:19:41.513585 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.513559 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:41.581703 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.581639 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sr8c\" (UniqueName: \"kubernetes.io/projected/f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa-kube-api-access-5sr8c\") pod \"authorino-b54df94b4-d5qfw\" (UID: \"f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa\") " pod="kuadrant-system/authorino-b54df94b4-d5qfw" Aug 04 18:19:41.591843 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.591816 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sr8c\" (UniqueName: \"kubernetes.io/projected/f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa-kube-api-access-5sr8c\") pod \"authorino-b54df94b4-d5qfw\" (UID: \"f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa\") " pod="kuadrant-system/authorino-b54df94b4-d5qfw" Aug 04 18:19:41.664078 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.664050 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-w9l5d"] Aug 04 18:19:41.667229 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.667210 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-w9l5d" Aug 04 18:19:41.683754 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.683731 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-d5qfw" Aug 04 18:19:41.685147 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.685122 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-w9l5d"] Aug 04 18:19:41.785690 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.785660 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsqrf\" (UniqueName: \"kubernetes.io/projected/f6209958-6d8b-4a92-99be-c2f37b452029-kube-api-access-rsqrf\") pod \"authorino-c548b5d7d-w9l5d\" (UID: \"f6209958-6d8b-4a92-99be-c2f37b452029\") " pod="kuadrant-system/authorino-c548b5d7d-w9l5d" Aug 04 18:19:41.815668 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.815636 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-d5qfw"] Aug 04 18:19:41.818209 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:19:41.818179 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9654d1e_a4d8_4b7a_bd30_fbbb4536cefa.slice/crio-eb3c927cda3b229a9120e2e240de54338aad70d68d12ded1db5d743a3d05a4ea WatchSource:0}: Error finding container eb3c927cda3b229a9120e2e240de54338aad70d68d12ded1db5d743a3d05a4ea: Status 404 returned error can't find the container with id eb3c927cda3b229a9120e2e240de54338aad70d68d12ded1db5d743a3d05a4ea Aug 04 18:19:41.886947 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.886874 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsqrf\" (UniqueName: \"kubernetes.io/projected/f6209958-6d8b-4a92-99be-c2f37b452029-kube-api-access-rsqrf\") pod \"authorino-c548b5d7d-w9l5d\" (UID: \"f6209958-6d8b-4a92-99be-c2f37b452029\") " pod="kuadrant-system/authorino-c548b5d7d-w9l5d" Aug 04 18:19:41.897648 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.897611 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsqrf\" (UniqueName: \"kubernetes.io/projected/f6209958-6d8b-4a92-99be-c2f37b452029-kube-api-access-rsqrf\") pod \"authorino-c548b5d7d-w9l5d\" (UID: \"f6209958-6d8b-4a92-99be-c2f37b452029\") " pod="kuadrant-system/authorino-c548b5d7d-w9l5d" Aug 04 18:19:41.954268 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.954230 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:41.976015 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:41.975994 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-w9l5d" Aug 04 18:19:42.112560 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:42.112526 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-w9l5d"] Aug 04 18:19:42.226475 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:19:42.226443 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6209958_6d8b_4a92_99be_c2f37b452029.slice/crio-baaf1f449d9e0a6a3a7d8da01f708b3878f3dfc3f49ebdce2ac04e6191c76534 WatchSource:0}: Error finding container baaf1f449d9e0a6a3a7d8da01f708b3878f3dfc3f49ebdce2ac04e6191c76534: Status 404 returned error can't find the container with id baaf1f449d9e0a6a3a7d8da01f708b3878f3dfc3f49ebdce2ac04e6191c76534 Aug 04 18:19:42.271754 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:42.271727 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-d5qfw" event={"ID":"f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa","Type":"ContainerStarted","Data":"eb3c927cda3b229a9120e2e240de54338aad70d68d12ded1db5d743a3d05a4ea"} Aug 04 18:19:42.272863 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:42.272841 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-w9l5d" event={"ID":"f6209958-6d8b-4a92-99be-c2f37b452029","Type":"ContainerStarted","Data":"baaf1f449d9e0a6a3a7d8da01f708b3878f3dfc3f49ebdce2ac04e6191c76534"} Aug 04 18:19:43.285261 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:43.285039 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" event={"ID":"103604c1-45a6-4eb5-bb86-71e0681f7197","Type":"ContainerStarted","Data":"43378cbb782e72ffa51e4d22090d4548ae5f5b18a6daa747973e8246178dae21"} Aug 04 18:19:43.285261 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:43.285107 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" Aug 04 18:19:43.304947 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:43.304608 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" podStartSLOduration=1.8659422970000001 podStartE2EDuration="3.304592148s" totalImagesPullingTime="1.438649851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:19:43.303400403 +0000 UTC m=+540.554682951" watchObservedRunningTime="2026-08-04 18:19:43.304592148 +0000 UTC m=+540.555874694" Aug 04 18:19:44.291035 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:44.290995 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-w9l5d" event={"ID":"f6209958-6d8b-4a92-99be-c2f37b452029","Type":"ContainerStarted","Data":"9613a56edbdfb857fc1b4254b8a5015c392c367855ced31745555eff6917ddbd"} Aug 04 18:19:44.292654 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:44.292624 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-d5qfw" event={"ID":"f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa","Type":"ContainerStarted","Data":"d96d6dafe8d02e18d8d682209bd3be40c0b118a92ae85c9745b35107bd5c25ca"} Aug 04 18:19:44.321080 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:44.321033 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-w9l5d" podStartSLOduration=1.431692343 podStartE2EDuration="3.321019849s" totalImagesPullingTime="1.889327506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:19:44.320086706 +0000 UTC m=+541.571369253" watchObservedRunningTime="2026-08-04 18:19:44.321019849 +0000 UTC m=+541.572302396" Aug 04 18:19:44.363597 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:44.363084 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-d5qfw" podStartSLOduration=1.055304816 podStartE2EDuration="3.363062358s" totalImagesPullingTime="2.307757542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:19:44.361947707 +0000 UTC m=+541.613230255" watchObservedRunningTime="2026-08-04 18:19:44.363062358 +0000 UTC m=+541.614344908" Aug 04 18:19:44.373650 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:44.373614 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-d5qfw"] Aug 04 18:19:44.786792 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:44.786756 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:44.887533 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:44.887500 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:46.301168 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:46.301127 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-d5qfw" podUID="f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa" containerName="authorino" containerID="cri-o://d96d6dafe8d02e18d8d682209bd3be40c0b118a92ae85c9745b35107bd5c25ca" gracePeriod=30 Aug 04 18:19:46.549654 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:46.549631 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-d5qfw" Aug 04 18:19:46.729469 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:46.729442 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sr8c\" (UniqueName: \"kubernetes.io/projected/f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa-kube-api-access-5sr8c\") pod \"f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa\" (UID: \"f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa\") " Aug 04 18:19:46.731461 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:46.731434 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa-kube-api-access-5sr8c" (OuterVolumeSpecName: "kube-api-access-5sr8c") pod "f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa" (UID: "f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa"). InnerVolumeSpecName "kube-api-access-5sr8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:19:46.830076 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:46.830045 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sr8c\" (UniqueName: \"kubernetes.io/projected/f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa-kube-api-access-5sr8c\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:19:47.254900 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:47.254872 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:19:47.306717 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:47.306690 2526 generic.go:356] "Generic (PLEG): container finished" podID="f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa" containerID="d96d6dafe8d02e18d8d682209bd3be40c0b118a92ae85c9745b35107bd5c25ca" exitCode=0 Aug 04 18:19:47.307101 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:47.306736 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-d5qfw" event={"ID":"f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa","Type":"ContainerDied","Data":"d96d6dafe8d02e18d8d682209bd3be40c0b118a92ae85c9745b35107bd5c25ca"} Aug 04 18:19:47.307101 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:47.306765 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-d5qfw" event={"ID":"f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa","Type":"ContainerDied","Data":"eb3c927cda3b229a9120e2e240de54338aad70d68d12ded1db5d743a3d05a4ea"} Aug 04 18:19:47.307101 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:47.306781 2526 scope.go:117] "RemoveContainer" containerID="d96d6dafe8d02e18d8d682209bd3be40c0b118a92ae85c9745b35107bd5c25ca" Aug 04 18:19:47.307101 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:47.306737 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-d5qfw" Aug 04 18:19:47.314690 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:47.314673 2526 scope.go:117] "RemoveContainer" containerID="d96d6dafe8d02e18d8d682209bd3be40c0b118a92ae85c9745b35107bd5c25ca" Aug 04 18:19:47.314939 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:19:47.314911 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d96d6dafe8d02e18d8d682209bd3be40c0b118a92ae85c9745b35107bd5c25ca\": container with ID starting with d96d6dafe8d02e18d8d682209bd3be40c0b118a92ae85c9745b35107bd5c25ca not found: ID does not exist" containerID="d96d6dafe8d02e18d8d682209bd3be40c0b118a92ae85c9745b35107bd5c25ca" Aug 04 18:19:47.315023 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:47.314941 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d96d6dafe8d02e18d8d682209bd3be40c0b118a92ae85c9745b35107bd5c25ca"} err="failed to get container status \"d96d6dafe8d02e18d8d682209bd3be40c0b118a92ae85c9745b35107bd5c25ca\": rpc error: code = NotFound desc = could not find container \"d96d6dafe8d02e18d8d682209bd3be40c0b118a92ae85c9745b35107bd5c25ca\": container with ID starting with d96d6dafe8d02e18d8d682209bd3be40c0b118a92ae85c9745b35107bd5c25ca not found: ID does not exist" Aug 04 18:19:47.326538 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:47.326514 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-d5qfw"] Aug 04 18:19:47.331307 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:47.331286 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-d5qfw"] Aug 04 18:19:49.180745 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:49.180712 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa" path="/var/lib/kubelet/pods/f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa/volumes" Aug 04 18:19:54.294159 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:54.294128 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" Aug 04 18:19:55.653651 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:55.653609 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-xnqpf"] Aug 04 18:19:55.654076 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:55.653782 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" podUID="103604c1-45a6-4eb5-bb86-71e0681f7197" containerName="limitador" containerID="cri-o://43378cbb782e72ffa51e4d22090d4548ae5f5b18a6daa747973e8246178dae21" gracePeriod=30 Aug 04 18:19:56.202499 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.202478 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" Aug 04 18:19:56.302849 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.302756 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/103604c1-45a6-4eb5-bb86-71e0681f7197-config-file\") pod \"103604c1-45a6-4eb5-bb86-71e0681f7197\" (UID: \"103604c1-45a6-4eb5-bb86-71e0681f7197\") " Aug 04 18:19:56.303051 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.302853 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldwvl\" (UniqueName: \"kubernetes.io/projected/103604c1-45a6-4eb5-bb86-71e0681f7197-kube-api-access-ldwvl\") pod \"103604c1-45a6-4eb5-bb86-71e0681f7197\" (UID: \"103604c1-45a6-4eb5-bb86-71e0681f7197\") " Aug 04 18:19:56.303157 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.303134 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103604c1-45a6-4eb5-bb86-71e0681f7197-config-file" (OuterVolumeSpecName: "config-file") pod "103604c1-45a6-4eb5-bb86-71e0681f7197" (UID: "103604c1-45a6-4eb5-bb86-71e0681f7197"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:19:56.304944 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.304914 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103604c1-45a6-4eb5-bb86-71e0681f7197-kube-api-access-ldwvl" (OuterVolumeSpecName: "kube-api-access-ldwvl") pod "103604c1-45a6-4eb5-bb86-71e0681f7197" (UID: "103604c1-45a6-4eb5-bb86-71e0681f7197"). InnerVolumeSpecName "kube-api-access-ldwvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:19:56.346878 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.346851 2526 generic.go:356] "Generic (PLEG): container finished" podID="103604c1-45a6-4eb5-bb86-71e0681f7197" containerID="43378cbb782e72ffa51e4d22090d4548ae5f5b18a6daa747973e8246178dae21" exitCode=0 Aug 04 18:19:56.347012 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.346893 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" Aug 04 18:19:56.347012 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.346933 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" event={"ID":"103604c1-45a6-4eb5-bb86-71e0681f7197","Type":"ContainerDied","Data":"43378cbb782e72ffa51e4d22090d4548ae5f5b18a6daa747973e8246178dae21"} Aug 04 18:19:56.347012 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.347001 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-xnqpf" event={"ID":"103604c1-45a6-4eb5-bb86-71e0681f7197","Type":"ContainerDied","Data":"966ddef9074cbc1da0ca02e19464b2074a29a35011815063142abfb008f80927"} Aug 04 18:19:56.347127 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.347024 2526 scope.go:117] "RemoveContainer" containerID="43378cbb782e72ffa51e4d22090d4548ae5f5b18a6daa747973e8246178dae21" Aug 04 18:19:56.355805 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.355789 2526 scope.go:117] "RemoveContainer" containerID="43378cbb782e72ffa51e4d22090d4548ae5f5b18a6daa747973e8246178dae21" Aug 04 18:19:56.356073 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:19:56.356055 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43378cbb782e72ffa51e4d22090d4548ae5f5b18a6daa747973e8246178dae21\": container with ID starting with 43378cbb782e72ffa51e4d22090d4548ae5f5b18a6daa747973e8246178dae21 not found: ID does not exist" containerID="43378cbb782e72ffa51e4d22090d4548ae5f5b18a6daa747973e8246178dae21" Aug 04 18:19:56.356124 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.356080 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43378cbb782e72ffa51e4d22090d4548ae5f5b18a6daa747973e8246178dae21"} err="failed to get container status \"43378cbb782e72ffa51e4d22090d4548ae5f5b18a6daa747973e8246178dae21\": rpc error: code = NotFound desc = could not find container \"43378cbb782e72ffa51e4d22090d4548ae5f5b18a6daa747973e8246178dae21\": container with ID starting with 43378cbb782e72ffa51e4d22090d4548ae5f5b18a6daa747973e8246178dae21 not found: ID does not exist" Aug 04 18:19:56.369086 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.369064 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-xnqpf"] Aug 04 18:19:56.373335 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.373317 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-xnqpf"] Aug 04 18:19:56.403923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.403902 2526 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/103604c1-45a6-4eb5-bb86-71e0681f7197-config-file\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:19:56.403923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:56.403920 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldwvl\" (UniqueName: \"kubernetes.io/projected/103604c1-45a6-4eb5-bb86-71e0681f7197-kube-api-access-ldwvl\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:19:57.180718 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:19:57.180683 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="103604c1-45a6-4eb5-bb86-71e0681f7197" path="/var/lib/kubelet/pods/103604c1-45a6-4eb5-bb86-71e0681f7197/volumes" Aug 04 18:20:01.779679 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:01.779643 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:01.946655 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:01.946617 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:02.266143 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.266117 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd"] Aug 04 18:20:02.266440 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.266429 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103604c1-45a6-4eb5-bb86-71e0681f7197" containerName="limitador" Aug 04 18:20:02.266484 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.266441 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="103604c1-45a6-4eb5-bb86-71e0681f7197" containerName="limitador" Aug 04 18:20:02.266484 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.266450 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa" containerName="authorino" Aug 04 18:20:02.266484 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.266455 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa" containerName="authorino" Aug 04 18:20:02.266574 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.266507 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9654d1e-a4d8-4b7a-bd30-fbbb4536cefa" containerName="authorino" Aug 04 18:20:02.266574 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.266518 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="103604c1-45a6-4eb5-bb86-71e0681f7197" containerName="limitador" Aug 04 18:20:02.270798 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.270782 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd" Aug 04 18:20:02.273860 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.273842 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:20:02.274138 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.274122 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Aug 04 18:20:02.274651 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.274634 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-4gf5v\"" Aug 04 18:20:02.274884 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.274872 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 18:20:02.278722 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.278700 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd"] Aug 04 18:20:02.313523 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.313497 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:02.344644 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.344622 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/89c68cc3-099d-4307-8375-4bcb9a5365b3-data\") pod \"postgres-69fb4594cf-g4hgd\" (UID: \"89c68cc3-099d-4307-8375-4bcb9a5365b3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd" Aug 04 18:20:02.344752 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.344653 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk2q4\" (UniqueName: \"kubernetes.io/projected/89c68cc3-099d-4307-8375-4bcb9a5365b3-kube-api-access-rk2q4\") pod \"postgres-69fb4594cf-g4hgd\" (UID: \"89c68cc3-099d-4307-8375-4bcb9a5365b3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd" Aug 04 18:20:02.446041 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.446001 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/89c68cc3-099d-4307-8375-4bcb9a5365b3-data\") pod \"postgres-69fb4594cf-g4hgd\" (UID: \"89c68cc3-099d-4307-8375-4bcb9a5365b3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd" Aug 04 18:20:02.446236 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.446054 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk2q4\" (UniqueName: \"kubernetes.io/projected/89c68cc3-099d-4307-8375-4bcb9a5365b3-kube-api-access-rk2q4\") pod \"postgres-69fb4594cf-g4hgd\" (UID: \"89c68cc3-099d-4307-8375-4bcb9a5365b3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd" Aug 04 18:20:02.446416 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.446393 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/89c68cc3-099d-4307-8375-4bcb9a5365b3-data\") pod \"postgres-69fb4594cf-g4hgd\" (UID: \"89c68cc3-099d-4307-8375-4bcb9a5365b3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd" Aug 04 18:20:02.456135 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.456114 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk2q4\" (UniqueName: \"kubernetes.io/projected/89c68cc3-099d-4307-8375-4bcb9a5365b3-kube-api-access-rk2q4\") pod \"postgres-69fb4594cf-g4hgd\" (UID: \"89c68cc3-099d-4307-8375-4bcb9a5365b3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd" Aug 04 18:20:02.582062 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.581972 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd" Aug 04 18:20:02.715395 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:02.715310 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd"] Aug 04 18:20:02.717788 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:20:02.717762 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c68cc3_099d_4307_8375_4bcb9a5365b3.slice/crio-c8b80ca40b6796ddc13098642da684d679309eb79cbfeb002ef22ef1d33546c1 WatchSource:0}: Error finding container c8b80ca40b6796ddc13098642da684d679309eb79cbfeb002ef22ef1d33546c1: Status 404 returned error can't find the container with id c8b80ca40b6796ddc13098642da684d679309eb79cbfeb002ef22ef1d33546c1 Aug 04 18:20:03.380121 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:03.380088 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd" event={"ID":"89c68cc3-099d-4307-8375-4bcb9a5365b3","Type":"ContainerStarted","Data":"c8b80ca40b6796ddc13098642da684d679309eb79cbfeb002ef22ef1d33546c1"} Aug 04 18:20:08.405003 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:08.404958 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd" event={"ID":"89c68cc3-099d-4307-8375-4bcb9a5365b3","Type":"ContainerStarted","Data":"df3afb2c92de853871529de0d1479ce562c5535f297901c04f86a7586cdbae01"} Aug 04 18:20:08.405390 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:08.405085 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd" Aug 04 18:20:08.423530 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:08.423484 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd" podStartSLOduration=1.5886836130000002 podStartE2EDuration="6.423470586s" totalImagesPullingTime="4.834786973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:20:08.42213359 +0000 UTC m=+565.673416136" watchObservedRunningTime="2026-08-04 18:20:08.423470586 +0000 UTC m=+565.674753133" Aug 04 18:20:14.438378 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:14.438349 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g4hgd" Aug 04 18:20:14.826117 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:14.826086 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:14.975834 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:14.975793 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:15.062308 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.062274 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-6lx56"] Aug 04 18:20:15.072642 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.072604 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-6lx56" Aug 04 18:20:15.073061 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.073033 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-6lx56"] Aug 04 18:20:15.123929 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.123905 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-6lx56"] Aug 04 18:20:15.124161 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:20:15.124143 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-8g8gq], unattached volumes=[], failed to process volumes=[kube-api-access-8g8gq]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-6lx56" podUID="ebd049e6-376f-4e44-aa1d-bfdc352d7a48" Aug 04 18:20:15.140849 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.140820 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g8gq\" (UniqueName: \"kubernetes.io/projected/ebd049e6-376f-4e44-aa1d-bfdc352d7a48-kube-api-access-8g8gq\") pod \"authorino-5dc7ccb5f-6lx56\" (UID: \"ebd049e6-376f-4e44-aa1d-bfdc352d7a48\") " pod="kuadrant-system/authorino-5dc7ccb5f-6lx56" Aug 04 18:20:15.153982 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.153941 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-59fhg"] Aug 04 18:20:15.157309 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.157290 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-59fhg" Aug 04 18:20:15.159665 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.159646 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 04 18:20:15.166693 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.166675 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-59fhg"] Aug 04 18:20:15.241454 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.241428 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kz5r\" (UniqueName: \"kubernetes.io/projected/44926125-5035-44d6-b8ea-0e9dd6b9e4a6-kube-api-access-9kz5r\") pod \"authorino-687bd45585-59fhg\" (UID: \"44926125-5035-44d6-b8ea-0e9dd6b9e4a6\") " pod="kuadrant-system/authorino-687bd45585-59fhg" Aug 04 18:20:15.241593 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.241476 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44926125-5035-44d6-b8ea-0e9dd6b9e4a6-tls-cert\") pod \"authorino-687bd45585-59fhg\" (UID: \"44926125-5035-44d6-b8ea-0e9dd6b9e4a6\") " pod="kuadrant-system/authorino-687bd45585-59fhg" Aug 04 18:20:15.241593 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.241509 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g8gq\" (UniqueName: \"kubernetes.io/projected/ebd049e6-376f-4e44-aa1d-bfdc352d7a48-kube-api-access-8g8gq\") pod \"authorino-5dc7ccb5f-6lx56\" (UID: \"ebd049e6-376f-4e44-aa1d-bfdc352d7a48\") " pod="kuadrant-system/authorino-5dc7ccb5f-6lx56" Aug 04 18:20:15.251080 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.251061 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g8gq\" (UniqueName: \"kubernetes.io/projected/ebd049e6-376f-4e44-aa1d-bfdc352d7a48-kube-api-access-8g8gq\") pod \"authorino-5dc7ccb5f-6lx56\" (UID: \"ebd049e6-376f-4e44-aa1d-bfdc352d7a48\") " pod="kuadrant-system/authorino-5dc7ccb5f-6lx56" Aug 04 18:20:15.324315 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.324289 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-59fhg"] Aug 04 18:20:15.324526 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:20:15.324509 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-9kz5r tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-59fhg" podUID="44926125-5035-44d6-b8ea-0e9dd6b9e4a6" Aug 04 18:20:15.342609 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.342545 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kz5r\" (UniqueName: \"kubernetes.io/projected/44926125-5035-44d6-b8ea-0e9dd6b9e4a6-kube-api-access-9kz5r\") pod \"authorino-687bd45585-59fhg\" (UID: \"44926125-5035-44d6-b8ea-0e9dd6b9e4a6\") " pod="kuadrant-system/authorino-687bd45585-59fhg" Aug 04 18:20:15.342609 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.342579 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44926125-5035-44d6-b8ea-0e9dd6b9e4a6-tls-cert\") pod \"authorino-687bd45585-59fhg\" (UID: \"44926125-5035-44d6-b8ea-0e9dd6b9e4a6\") " pod="kuadrant-system/authorino-687bd45585-59fhg" Aug 04 18:20:15.345633 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.345614 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44926125-5035-44d6-b8ea-0e9dd6b9e4a6-tls-cert\") pod \"authorino-687bd45585-59fhg\" (UID: \"44926125-5035-44d6-b8ea-0e9dd6b9e4a6\") " pod="kuadrant-system/authorino-687bd45585-59fhg" Aug 04 18:20:15.352783 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.352760 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kz5r\" (UniqueName: \"kubernetes.io/projected/44926125-5035-44d6-b8ea-0e9dd6b9e4a6-kube-api-access-9kz5r\") pod \"authorino-687bd45585-59fhg\" (UID: \"44926125-5035-44d6-b8ea-0e9dd6b9e4a6\") " pod="kuadrant-system/authorino-687bd45585-59fhg" Aug 04 18:20:15.435908 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.435884 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-6lx56" Aug 04 18:20:15.436036 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.435884 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-59fhg" Aug 04 18:20:15.443536 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.443516 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-6lx56" Aug 04 18:20:15.446159 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.446141 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-59fhg" Aug 04 18:20:15.544208 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.544187 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kz5r\" (UniqueName: \"kubernetes.io/projected/44926125-5035-44d6-b8ea-0e9dd6b9e4a6-kube-api-access-9kz5r\") pod \"44926125-5035-44d6-b8ea-0e9dd6b9e4a6\" (UID: \"44926125-5035-44d6-b8ea-0e9dd6b9e4a6\") " Aug 04 18:20:15.544317 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.544251 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g8gq\" (UniqueName: \"kubernetes.io/projected/ebd049e6-376f-4e44-aa1d-bfdc352d7a48-kube-api-access-8g8gq\") pod \"ebd049e6-376f-4e44-aa1d-bfdc352d7a48\" (UID: \"ebd049e6-376f-4e44-aa1d-bfdc352d7a48\") " Aug 04 18:20:15.544317 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.544270 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44926125-5035-44d6-b8ea-0e9dd6b9e4a6-tls-cert\") pod \"44926125-5035-44d6-b8ea-0e9dd6b9e4a6\" (UID: \"44926125-5035-44d6-b8ea-0e9dd6b9e4a6\") " Aug 04 18:20:15.546326 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.546298 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd049e6-376f-4e44-aa1d-bfdc352d7a48-kube-api-access-8g8gq" (OuterVolumeSpecName: "kube-api-access-8g8gq") pod "ebd049e6-376f-4e44-aa1d-bfdc352d7a48" (UID: "ebd049e6-376f-4e44-aa1d-bfdc352d7a48"). InnerVolumeSpecName "kube-api-access-8g8gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:20:15.546326 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.546318 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44926125-5035-44d6-b8ea-0e9dd6b9e4a6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "44926125-5035-44d6-b8ea-0e9dd6b9e4a6" (UID: "44926125-5035-44d6-b8ea-0e9dd6b9e4a6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:20:15.546447 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.546341 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44926125-5035-44d6-b8ea-0e9dd6b9e4a6-kube-api-access-9kz5r" (OuterVolumeSpecName: "kube-api-access-9kz5r") pod "44926125-5035-44d6-b8ea-0e9dd6b9e4a6" (UID: "44926125-5035-44d6-b8ea-0e9dd6b9e4a6"). InnerVolumeSpecName "kube-api-access-9kz5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:20:15.645506 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.645447 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g8gq\" (UniqueName: \"kubernetes.io/projected/ebd049e6-376f-4e44-aa1d-bfdc352d7a48-kube-api-access-8g8gq\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:20:15.645506 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.645479 2526 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44926125-5035-44d6-b8ea-0e9dd6b9e4a6-tls-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:20:15.645506 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:15.645490 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kz5r\" (UniqueName: \"kubernetes.io/projected/44926125-5035-44d6-b8ea-0e9dd6b9e4a6-kube-api-access-9kz5r\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:20:16.440019 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:16.439987 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-6lx56" Aug 04 18:20:16.440178 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:16.439987 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-59fhg" Aug 04 18:20:16.473872 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:16.473846 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-6lx56"] Aug 04 18:20:16.478479 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:16.478458 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-6lx56"] Aug 04 18:20:16.503880 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:16.503857 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-59fhg"] Aug 04 18:20:16.506151 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:16.506128 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-59fhg"] Aug 04 18:20:17.180936 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:17.180904 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44926125-5035-44d6-b8ea-0e9dd6b9e4a6" path="/var/lib/kubelet/pods/44926125-5035-44d6-b8ea-0e9dd6b9e4a6/volumes" Aug 04 18:20:17.181225 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:17.181206 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd049e6-376f-4e44-aa1d-bfdc352d7a48" path="/var/lib/kubelet/pods/ebd049e6-376f-4e44-aa1d-bfdc352d7a48/volumes" Aug 04 18:20:18.588487 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:18.588456 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:18.663319 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:18.663291 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:18.803720 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:18.803694 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-w9l5d"] Aug 04 18:20:18.803862 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:18.803842 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-w9l5d" podUID="f6209958-6d8b-4a92-99be-c2f37b452029" containerName="authorino" containerID="cri-o://9613a56edbdfb857fc1b4254b8a5015c392c367855ced31745555eff6917ddbd" gracePeriod=30 Aug 04 18:20:19.051066 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:19.051043 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-w9l5d" Aug 04 18:20:19.175133 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:19.175071 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsqrf\" (UniqueName: \"kubernetes.io/projected/f6209958-6d8b-4a92-99be-c2f37b452029-kube-api-access-rsqrf\") pod \"f6209958-6d8b-4a92-99be-c2f37b452029\" (UID: \"f6209958-6d8b-4a92-99be-c2f37b452029\") " Aug 04 18:20:19.184118 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:19.184069 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6209958-6d8b-4a92-99be-c2f37b452029-kube-api-access-rsqrf" (OuterVolumeSpecName: "kube-api-access-rsqrf") pod "f6209958-6d8b-4a92-99be-c2f37b452029" (UID: "f6209958-6d8b-4a92-99be-c2f37b452029"). InnerVolumeSpecName "kube-api-access-rsqrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:20:19.275649 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:19.275621 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsqrf\" (UniqueName: \"kubernetes.io/projected/f6209958-6d8b-4a92-99be-c2f37b452029-kube-api-access-rsqrf\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:20:19.454380 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:19.454354 2526 generic.go:356] "Generic (PLEG): container finished" podID="f6209958-6d8b-4a92-99be-c2f37b452029" containerID="9613a56edbdfb857fc1b4254b8a5015c392c367855ced31745555eff6917ddbd" exitCode=0 Aug 04 18:20:19.454523 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:19.454412 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-w9l5d" Aug 04 18:20:19.454523 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:19.454429 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-w9l5d" event={"ID":"f6209958-6d8b-4a92-99be-c2f37b452029","Type":"ContainerDied","Data":"9613a56edbdfb857fc1b4254b8a5015c392c367855ced31745555eff6917ddbd"} Aug 04 18:20:19.454523 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:19.454467 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-w9l5d" event={"ID":"f6209958-6d8b-4a92-99be-c2f37b452029","Type":"ContainerDied","Data":"baaf1f449d9e0a6a3a7d8da01f708b3878f3dfc3f49ebdce2ac04e6191c76534"} Aug 04 18:20:19.454523 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:19.454484 2526 scope.go:117] "RemoveContainer" containerID="9613a56edbdfb857fc1b4254b8a5015c392c367855ced31745555eff6917ddbd" Aug 04 18:20:19.462385 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:19.462367 2526 scope.go:117] "RemoveContainer" containerID="9613a56edbdfb857fc1b4254b8a5015c392c367855ced31745555eff6917ddbd" Aug 04 18:20:19.462643 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:20:19.462618 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9613a56edbdfb857fc1b4254b8a5015c392c367855ced31745555eff6917ddbd\": container with ID starting with 9613a56edbdfb857fc1b4254b8a5015c392c367855ced31745555eff6917ddbd not found: ID does not exist" containerID="9613a56edbdfb857fc1b4254b8a5015c392c367855ced31745555eff6917ddbd" Aug 04 18:20:19.462736 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:19.462652 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9613a56edbdfb857fc1b4254b8a5015c392c367855ced31745555eff6917ddbd"} err="failed to get container status \"9613a56edbdfb857fc1b4254b8a5015c392c367855ced31745555eff6917ddbd\": rpc error: code = NotFound desc = could not find container \"9613a56edbdfb857fc1b4254b8a5015c392c367855ced31745555eff6917ddbd\": container with ID starting with 9613a56edbdfb857fc1b4254b8a5015c392c367855ced31745555eff6917ddbd not found: ID does not exist" Aug 04 18:20:19.473219 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:19.473198 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-w9l5d"] Aug 04 18:20:19.477911 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:19.477891 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-w9l5d"] Aug 04 18:20:21.180487 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:21.180454 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6209958-6d8b-4a92-99be-c2f37b452029" path="/var/lib/kubelet/pods/f6209958-6d8b-4a92-99be-c2f37b452029/volumes" Aug 04 18:20:23.860798 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:23.860762 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:23.862023 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:23.862003 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cfd6b4d4-z7d6s"] Aug 04 18:20:23.862473 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:23.862459 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6209958-6d8b-4a92-99be-c2f37b452029" containerName="authorino" Aug 04 18:20:23.862525 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:23.862476 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6209958-6d8b-4a92-99be-c2f37b452029" containerName="authorino" Aug 04 18:20:23.862573 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:23.862555 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6209958-6d8b-4a92-99be-c2f37b452029" containerName="authorino" Aug 04 18:20:23.868038 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:23.868018 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" Aug 04 18:20:23.873355 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:23.873333 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j4d7j\"" Aug 04 18:20:23.873630 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:23.873563 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 18:20:23.874174 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:23.874158 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 18:20:23.894154 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:23.894125 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cfd6b4d4-z7d6s"] Aug 04 18:20:23.910814 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:23.910787 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gn27\" (UniqueName: \"kubernetes.io/projected/f7a06308-bf5f-4d22-9221-1540c5e187c3-kube-api-access-6gn27\") pod \"maas-controller-cfd6b4d4-z7d6s\" (UID: \"f7a06308-bf5f-4d22-9221-1540c5e187c3\") " pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" Aug 04 18:20:23.910936 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:23.910820 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7a06308-bf5f-4d22-9221-1540c5e187c3-webhook-cert\") pod \"maas-controller-cfd6b4d4-z7d6s\" (UID: \"f7a06308-bf5f-4d22-9221-1540c5e187c3\") " pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" Aug 04 18:20:23.974977 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:23.974937 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:24.011341 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.011316 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gn27\" (UniqueName: \"kubernetes.io/projected/f7a06308-bf5f-4d22-9221-1540c5e187c3-kube-api-access-6gn27\") pod \"maas-controller-cfd6b4d4-z7d6s\" (UID: \"f7a06308-bf5f-4d22-9221-1540c5e187c3\") " pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" Aug 04 18:20:24.011447 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.011358 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7a06308-bf5f-4d22-9221-1540c5e187c3-webhook-cert\") pod \"maas-controller-cfd6b4d4-z7d6s\" (UID: \"f7a06308-bf5f-4d22-9221-1540c5e187c3\") " pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" Aug 04 18:20:24.011495 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:20:24.011470 2526 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Aug 04 18:20:24.011552 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:20:24.011542 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7a06308-bf5f-4d22-9221-1540c5e187c3-webhook-cert podName:f7a06308-bf5f-4d22-9221-1540c5e187c3 nodeName:}" failed. No retries permitted until 2026-08-04 18:20:24.51152226 +0000 UTC m=+581.762804790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/f7a06308-bf5f-4d22-9221-1540c5e187c3-webhook-cert") pod "maas-controller-cfd6b4d4-z7d6s" (UID: "f7a06308-bf5f-4d22-9221-1540c5e187c3") : secret "maas-controller-webhook-cert" not found Aug 04 18:20:24.021217 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.021189 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gn27\" (UniqueName: \"kubernetes.io/projected/f7a06308-bf5f-4d22-9221-1540c5e187c3-kube-api-access-6gn27\") pod \"maas-controller-cfd6b4d4-z7d6s\" (UID: \"f7a06308-bf5f-4d22-9221-1540c5e187c3\") " pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" Aug 04 18:20:24.043130 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.043109 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cffffc5dc-2zdrb"] Aug 04 18:20:24.046760 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.046746 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" Aug 04 18:20:24.057766 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.057747 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cffffc5dc-2zdrb"] Aug 04 18:20:24.112755 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.112695 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed612c71-3187-43f9-bcea-bdfd39590463-webhook-cert\") pod \"maas-controller-5cffffc5dc-2zdrb\" (UID: \"ed612c71-3187-43f9-bcea-bdfd39590463\") " pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" Aug 04 18:20:24.112755 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.112727 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9zmm\" (UniqueName: \"kubernetes.io/projected/ed612c71-3187-43f9-bcea-bdfd39590463-kube-api-access-b9zmm\") pod \"maas-controller-5cffffc5dc-2zdrb\" (UID: \"ed612c71-3187-43f9-bcea-bdfd39590463\") " pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" Aug 04 18:20:24.213920 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.213889 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed612c71-3187-43f9-bcea-bdfd39590463-webhook-cert\") pod \"maas-controller-5cffffc5dc-2zdrb\" (UID: \"ed612c71-3187-43f9-bcea-bdfd39590463\") " pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" Aug 04 18:20:24.214075 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.213926 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9zmm\" (UniqueName: \"kubernetes.io/projected/ed612c71-3187-43f9-bcea-bdfd39590463-kube-api-access-b9zmm\") pod \"maas-controller-5cffffc5dc-2zdrb\" (UID: \"ed612c71-3187-43f9-bcea-bdfd39590463\") " pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" Aug 04 18:20:24.214075 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:20:24.214056 2526 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Aug 04 18:20:24.214148 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:20:24.214118 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed612c71-3187-43f9-bcea-bdfd39590463-webhook-cert podName:ed612c71-3187-43f9-bcea-bdfd39590463 nodeName:}" failed. No retries permitted until 2026-08-04 18:20:24.714101201 +0000 UTC m=+581.965383735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/ed612c71-3187-43f9-bcea-bdfd39590463-webhook-cert") pod "maas-controller-5cffffc5dc-2zdrb" (UID: "ed612c71-3187-43f9-bcea-bdfd39590463") : secret "maas-controller-webhook-cert" not found Aug 04 18:20:24.226162 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.226136 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9zmm\" (UniqueName: \"kubernetes.io/projected/ed612c71-3187-43f9-bcea-bdfd39590463-kube-api-access-b9zmm\") pod \"maas-controller-5cffffc5dc-2zdrb\" (UID: \"ed612c71-3187-43f9-bcea-bdfd39590463\") " pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" Aug 04 18:20:24.314948 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.314912 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:24.515955 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.515933 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7a06308-bf5f-4d22-9221-1540c5e187c3-webhook-cert\") pod \"maas-controller-cfd6b4d4-z7d6s\" (UID: \"f7a06308-bf5f-4d22-9221-1540c5e187c3\") " pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" Aug 04 18:20:24.518188 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.518161 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7a06308-bf5f-4d22-9221-1540c5e187c3-webhook-cert\") pod \"maas-controller-cfd6b4d4-z7d6s\" (UID: \"f7a06308-bf5f-4d22-9221-1540c5e187c3\") " pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" Aug 04 18:20:24.717438 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.717410 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed612c71-3187-43f9-bcea-bdfd39590463-webhook-cert\") pod \"maas-controller-5cffffc5dc-2zdrb\" (UID: \"ed612c71-3187-43f9-bcea-bdfd39590463\") " pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" Aug 04 18:20:24.719544 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.719528 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed612c71-3187-43f9-bcea-bdfd39590463-webhook-cert\") pod \"maas-controller-5cffffc5dc-2zdrb\" (UID: \"ed612c71-3187-43f9-bcea-bdfd39590463\") " pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" Aug 04 18:20:24.778622 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.778562 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" Aug 04 18:20:24.904488 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.904465 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cfd6b4d4-z7d6s"] Aug 04 18:20:24.906857 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:20:24.906830 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a06308_bf5f_4d22_9221_1540c5e187c3.slice/crio-a728fbcdf0e7c871bb69fe629e9904469c793cecfb54f11655515744c90e9205 WatchSource:0}: Error finding container a728fbcdf0e7c871bb69fe629e9904469c793cecfb54f11655515744c90e9205: Status 404 returned error can't find the container with id a728fbcdf0e7c871bb69fe629e9904469c793cecfb54f11655515744c90e9205 Aug 04 18:20:24.958860 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:24.958837 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" Aug 04 18:20:25.091840 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:25.091816 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cffffc5dc-2zdrb"] Aug 04 18:20:25.093936 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:20:25.093908 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded612c71_3187_43f9_bcea_bdfd39590463.slice/crio-de1d5bcab2599821cb032ae32ba3b3d6de23b86f1b767dd6e8a30bd2c86155ff WatchSource:0}: Error finding container de1d5bcab2599821cb032ae32ba3b3d6de23b86f1b767dd6e8a30bd2c86155ff: Status 404 returned error can't find the container with id de1d5bcab2599821cb032ae32ba3b3d6de23b86f1b767dd6e8a30bd2c86155ff Aug 04 18:20:25.484176 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:25.484147 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" event={"ID":"ed612c71-3187-43f9-bcea-bdfd39590463","Type":"ContainerStarted","Data":"de1d5bcab2599821cb032ae32ba3b3d6de23b86f1b767dd6e8a30bd2c86155ff"} Aug 04 18:20:25.485098 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:25.485076 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" event={"ID":"f7a06308-bf5f-4d22-9221-1540c5e187c3","Type":"ContainerStarted","Data":"a728fbcdf0e7c871bb69fe629e9904469c793cecfb54f11655515744c90e9205"} Aug 04 18:20:27.444904 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:27.444873 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:27.496231 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:27.496201 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" event={"ID":"f7a06308-bf5f-4d22-9221-1540c5e187c3","Type":"ContainerStarted","Data":"6f49bb6ed3cd24f680e4de4caa30b131db1cc62cad45d99086a386c59fe66a95"} Aug 04 18:20:27.496387 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:27.496274 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" Aug 04 18:20:27.497626 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:27.497602 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" event={"ID":"ed612c71-3187-43f9-bcea-bdfd39590463","Type":"ContainerStarted","Data":"b2aa3647cc5637f9f2d917e4463aec2b61e6101e3a2707d99865dd120a0c12f2"} Aug 04 18:20:27.497728 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:27.497653 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" Aug 04 18:20:27.535132 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:27.535059 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" podStartSLOduration=2.178655164 podStartE2EDuration="4.535045542s" totalImagesPullingTime="2.356390378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:20:27.529985456 +0000 UTC m=+584.781267993" watchObservedRunningTime="2026-08-04 18:20:27.535045542 +0000 UTC m=+584.786328113" Aug 04 18:20:27.560645 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:27.560607 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" podStartSLOduration=1.3910788680000001 podStartE2EDuration="3.560597259s" totalImagesPullingTime="2.169518391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:20:27.559500003 +0000 UTC m=+584.810782550" watchObservedRunningTime="2026-08-04 18:20:27.560597259 +0000 UTC m=+584.811879835" Aug 04 18:20:27.589834 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:27.589804 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:38.503649 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:38.503619 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" Aug 04 18:20:38.504041 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:38.503686 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" Aug 04 18:20:38.572728 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:38.572694 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cfd6b4d4-z7d6s"] Aug 04 18:20:38.572935 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:38.572905 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" podUID="f7a06308-bf5f-4d22-9221-1540c5e187c3" containerName="manager" containerID="cri-o://6f49bb6ed3cd24f680e4de4caa30b131db1cc62cad45d99086a386c59fe66a95" gracePeriod=30 Aug 04 18:20:38.830977 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:38.830938 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" Aug 04 18:20:38.939820 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:38.939793 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gn27\" (UniqueName: \"kubernetes.io/projected/f7a06308-bf5f-4d22-9221-1540c5e187c3-kube-api-access-6gn27\") pod \"f7a06308-bf5f-4d22-9221-1540c5e187c3\" (UID: \"f7a06308-bf5f-4d22-9221-1540c5e187c3\") " Aug 04 18:20:38.940000 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:38.939880 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7a06308-bf5f-4d22-9221-1540c5e187c3-webhook-cert\") pod \"f7a06308-bf5f-4d22-9221-1540c5e187c3\" (UID: \"f7a06308-bf5f-4d22-9221-1540c5e187c3\") " Aug 04 18:20:38.941959 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:38.941932 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7a06308-bf5f-4d22-9221-1540c5e187c3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f7a06308-bf5f-4d22-9221-1540c5e187c3" (UID: "f7a06308-bf5f-4d22-9221-1540c5e187c3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:20:38.942075 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:38.942012 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a06308-bf5f-4d22-9221-1540c5e187c3-kube-api-access-6gn27" (OuterVolumeSpecName: "kube-api-access-6gn27") pod "f7a06308-bf5f-4d22-9221-1540c5e187c3" (UID: "f7a06308-bf5f-4d22-9221-1540c5e187c3"). InnerVolumeSpecName "kube-api-access-6gn27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:20:39.040767 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:39.040743 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gn27\" (UniqueName: \"kubernetes.io/projected/f7a06308-bf5f-4d22-9221-1540c5e187c3-kube-api-access-6gn27\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:20:39.040767 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:39.040766 2526 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7a06308-bf5f-4d22-9221-1540c5e187c3-webhook-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:20:39.557617 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:39.557586 2526 generic.go:356] "Generic (PLEG): container finished" podID="f7a06308-bf5f-4d22-9221-1540c5e187c3" containerID="6f49bb6ed3cd24f680e4de4caa30b131db1cc62cad45d99086a386c59fe66a95" exitCode=0 Aug 04 18:20:39.558052 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:39.557629 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" Aug 04 18:20:39.558052 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:39.557638 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" event={"ID":"f7a06308-bf5f-4d22-9221-1540c5e187c3","Type":"ContainerDied","Data":"6f49bb6ed3cd24f680e4de4caa30b131db1cc62cad45d99086a386c59fe66a95"} Aug 04 18:20:39.558052 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:39.557692 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cfd6b4d4-z7d6s" event={"ID":"f7a06308-bf5f-4d22-9221-1540c5e187c3","Type":"ContainerDied","Data":"a728fbcdf0e7c871bb69fe629e9904469c793cecfb54f11655515744c90e9205"} Aug 04 18:20:39.558052 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:39.557724 2526 scope.go:117] "RemoveContainer" containerID="6f49bb6ed3cd24f680e4de4caa30b131db1cc62cad45d99086a386c59fe66a95" Aug 04 18:20:39.565747 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:39.565736 2526 scope.go:117] "RemoveContainer" containerID="6f49bb6ed3cd24f680e4de4caa30b131db1cc62cad45d99086a386c59fe66a95" Aug 04 18:20:39.566050 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:20:39.566031 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f49bb6ed3cd24f680e4de4caa30b131db1cc62cad45d99086a386c59fe66a95\": container with ID starting with 6f49bb6ed3cd24f680e4de4caa30b131db1cc62cad45d99086a386c59fe66a95 not found: ID does not exist" containerID="6f49bb6ed3cd24f680e4de4caa30b131db1cc62cad45d99086a386c59fe66a95" Aug 04 18:20:39.566101 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:39.566060 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f49bb6ed3cd24f680e4de4caa30b131db1cc62cad45d99086a386c59fe66a95"} err="failed to get container status \"6f49bb6ed3cd24f680e4de4caa30b131db1cc62cad45d99086a386c59fe66a95\": rpc error: code = NotFound desc = could not find container \"6f49bb6ed3cd24f680e4de4caa30b131db1cc62cad45d99086a386c59fe66a95\": container with ID starting with 6f49bb6ed3cd24f680e4de4caa30b131db1cc62cad45d99086a386c59fe66a95 not found: ID does not exist" Aug 04 18:20:39.575273 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:39.575253 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cfd6b4d4-z7d6s"] Aug 04 18:20:39.583066 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:39.583047 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cfd6b4d4-z7d6s"] Aug 04 18:20:41.181600 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:41.181567 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7a06308-bf5f-4d22-9221-1540c5e187c3" path="/var/lib/kubelet/pods/f7a06308-bf5f-4d22-9221-1540c5e187c3/volumes" Aug 04 18:20:43.116239 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:43.116197 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:43.192865 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:43.192839 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:59.003795 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:59.003757 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:59.084157 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:59.084123 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:59.229074 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:59.229030 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:20:59.334158 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:20:59.334082 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:00.865604 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:00.865573 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:00.922291 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:00.922263 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb"] Aug 04 18:21:00.922733 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:00.922720 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7a06308-bf5f-4d22-9221-1540c5e187c3" containerName="manager" Aug 04 18:21:00.922792 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:00.922736 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7a06308-bf5f-4d22-9221-1540c5e187c3" containerName="manager" Aug 04 18:21:00.922848 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:00.922838 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7a06308-bf5f-4d22-9221-1540c5e187c3" containerName="manager" Aug 04 18:21:00.937671 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:00.937639 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb"] Aug 04 18:21:00.937790 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:00.937769 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" Aug 04 18:21:00.941756 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:00.941732 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Aug 04 18:21:00.941882 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:00.941816 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bh448\"" Aug 04 18:21:01.107690 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.107657 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45be4bd-bec4-438e-a844-4b8df8730422-openshift-service-ca\") pod \"maas-api-846c96f86f-fs6jb\" (UID: \"d45be4bd-bec4-438e-a844-4b8df8730422\") " pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" Aug 04 18:21:01.107827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.107771 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42d5r\" (UniqueName: \"kubernetes.io/projected/d45be4bd-bec4-438e-a844-4b8df8730422-kube-api-access-42d5r\") pod \"maas-api-846c96f86f-fs6jb\" (UID: \"d45be4bd-bec4-438e-a844-4b8df8730422\") " pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" Aug 04 18:21:01.107827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.107806 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d45be4bd-bec4-438e-a844-4b8df8730422-maas-api-tls\") pod \"maas-api-846c96f86f-fs6jb\" (UID: \"d45be4bd-bec4-438e-a844-4b8df8730422\") " pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" Aug 04 18:21:01.204352 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.204322 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:01.208372 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.208351 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42d5r\" (UniqueName: \"kubernetes.io/projected/d45be4bd-bec4-438e-a844-4b8df8730422-kube-api-access-42d5r\") pod \"maas-api-846c96f86f-fs6jb\" (UID: \"d45be4bd-bec4-438e-a844-4b8df8730422\") " pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" Aug 04 18:21:01.208435 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.208388 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d45be4bd-bec4-438e-a844-4b8df8730422-maas-api-tls\") pod \"maas-api-846c96f86f-fs6jb\" (UID: \"d45be4bd-bec4-438e-a844-4b8df8730422\") " pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" Aug 04 18:21:01.208435 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.208409 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45be4bd-bec4-438e-a844-4b8df8730422-openshift-service-ca\") pod \"maas-api-846c96f86f-fs6jb\" (UID: \"d45be4bd-bec4-438e-a844-4b8df8730422\") " pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" Aug 04 18:21:01.208948 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.208928 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45be4bd-bec4-438e-a844-4b8df8730422-openshift-service-ca\") pod \"maas-api-846c96f86f-fs6jb\" (UID: \"d45be4bd-bec4-438e-a844-4b8df8730422\") " pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" Aug 04 18:21:01.210681 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.210666 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d45be4bd-bec4-438e-a844-4b8df8730422-maas-api-tls\") pod \"maas-api-846c96f86f-fs6jb\" (UID: \"d45be4bd-bec4-438e-a844-4b8df8730422\") " pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" Aug 04 18:21:01.219486 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.219467 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42d5r\" (UniqueName: \"kubernetes.io/projected/d45be4bd-bec4-438e-a844-4b8df8730422-kube-api-access-42d5r\") pod \"maas-api-846c96f86f-fs6jb\" (UID: \"d45be4bd-bec4-438e-a844-4b8df8730422\") " pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" Aug 04 18:21:01.250490 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.250465 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" Aug 04 18:21:01.390694 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.390500 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb"] Aug 04 18:21:01.393166 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:21:01.393139 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45be4bd_bec4_438e_a844_4b8df8730422.slice/crio-fd20cfe5231a7c2f1eabc8cbd52518b217384cc0a22ed4322fec2ada16113279 WatchSource:0}: Error finding container fd20cfe5231a7c2f1eabc8cbd52518b217384cc0a22ed4322fec2ada16113279: Status 404 returned error can't find the container with id fd20cfe5231a7c2f1eabc8cbd52518b217384cc0a22ed4322fec2ada16113279 Aug 04 18:21:01.432635 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.432611 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:01.531681 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.531595 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:01.568692 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.568660 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-c9kjx"] Aug 04 18:21:01.576888 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.576870 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-c9kjx" Aug 04 18:21:01.580011 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.579991 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Aug 04 18:21:01.580118 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.580065 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-flpvg\"" Aug 04 18:21:01.583625 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.583598 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-c9kjx"] Aug 04 18:21:01.635484 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.635460 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf"] Aug 04 18:21:01.640138 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.640123 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf" Aug 04 18:21:01.661258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.650709 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf"] Aug 04 18:21:01.661258 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.660328 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" event={"ID":"d45be4bd-bec4-438e-a844-4b8df8730422","Type":"ContainerStarted","Data":"fd20cfe5231a7c2f1eabc8cbd52518b217384cc0a22ed4322fec2ada16113279"} Aug 04 18:21:01.711811 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.711767 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fq5m\" (UniqueName: \"kubernetes.io/projected/3346ba4e-c7fa-4d34-83f7-2780f3b42d33-kube-api-access-8fq5m\") pod \"payload-processing-5cc6cd444b-c9kjx\" (UID: \"3346ba4e-c7fa-4d34-83f7-2780f3b42d33\") " pod="openshift-ingress/payload-processing-5cc6cd444b-c9kjx" Aug 04 18:21:01.711947 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.711864 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3346ba4e-c7fa-4d34-83f7-2780f3b42d33-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-c9kjx\" (UID: \"3346ba4e-c7fa-4d34-83f7-2780f3b42d33\") " pod="openshift-ingress/payload-processing-5cc6cd444b-c9kjx" Aug 04 18:21:01.813186 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.813096 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3346ba4e-c7fa-4d34-83f7-2780f3b42d33-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-c9kjx\" (UID: \"3346ba4e-c7fa-4d34-83f7-2780f3b42d33\") " pod="openshift-ingress/payload-processing-5cc6cd444b-c9kjx" Aug 04 18:21:01.813348 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.813198 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426116a3-0958-4362-89e3-b9900c902faa-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-zftxf\" (UID: \"426116a3-0958-4362-89e3-b9900c902faa\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf" Aug 04 18:21:01.813348 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.813238 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crgpv\" (UniqueName: \"kubernetes.io/projected/426116a3-0958-4362-89e3-b9900c902faa-kube-api-access-crgpv\") pod \"payload-pre-processing-687dbb6cf6-zftxf\" (UID: \"426116a3-0958-4362-89e3-b9900c902faa\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf" Aug 04 18:21:01.813348 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.813317 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fq5m\" (UniqueName: \"kubernetes.io/projected/3346ba4e-c7fa-4d34-83f7-2780f3b42d33-kube-api-access-8fq5m\") pod \"payload-processing-5cc6cd444b-c9kjx\" (UID: \"3346ba4e-c7fa-4d34-83f7-2780f3b42d33\") " pod="openshift-ingress/payload-processing-5cc6cd444b-c9kjx" Aug 04 18:21:01.813788 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.813764 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3346ba4e-c7fa-4d34-83f7-2780f3b42d33-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-c9kjx\" (UID: \"3346ba4e-c7fa-4d34-83f7-2780f3b42d33\") " pod="openshift-ingress/payload-processing-5cc6cd444b-c9kjx" Aug 04 18:21:01.823657 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.823633 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fq5m\" (UniqueName: \"kubernetes.io/projected/3346ba4e-c7fa-4d34-83f7-2780f3b42d33-kube-api-access-8fq5m\") pod \"payload-processing-5cc6cd444b-c9kjx\" (UID: \"3346ba4e-c7fa-4d34-83f7-2780f3b42d33\") " pod="openshift-ingress/payload-processing-5cc6cd444b-c9kjx" Aug 04 18:21:01.888786 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.888756 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-c9kjx" Aug 04 18:21:01.914180 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.914150 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426116a3-0958-4362-89e3-b9900c902faa-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-zftxf\" (UID: \"426116a3-0958-4362-89e3-b9900c902faa\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf" Aug 04 18:21:01.914321 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.914192 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crgpv\" (UniqueName: \"kubernetes.io/projected/426116a3-0958-4362-89e3-b9900c902faa-kube-api-access-crgpv\") pod \"payload-pre-processing-687dbb6cf6-zftxf\" (UID: \"426116a3-0958-4362-89e3-b9900c902faa\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf" Aug 04 18:21:01.914856 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.914831 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426116a3-0958-4362-89e3-b9900c902faa-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-zftxf\" (UID: \"426116a3-0958-4362-89e3-b9900c902faa\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf" Aug 04 18:21:01.923736 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.923713 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crgpv\" (UniqueName: \"kubernetes.io/projected/426116a3-0958-4362-89e3-b9900c902faa-kube-api-access-crgpv\") pod \"payload-pre-processing-687dbb6cf6-zftxf\" (UID: \"426116a3-0958-4362-89e3-b9900c902faa\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf" Aug 04 18:21:01.949844 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:01.949818 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf" Aug 04 18:21:02.037564 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:02.037509 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-c9kjx"] Aug 04 18:21:02.040054 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:21:02.040020 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3346ba4e_c7fa_4d34_83f7_2780f3b42d33.slice/crio-c1c9a567c24093c8948f7b651e7e60045bb101b44f13f9d81a1a0e88453b42d9 WatchSource:0}: Error finding container c1c9a567c24093c8948f7b651e7e60045bb101b44f13f9d81a1a0e88453b42d9: Status 404 returned error can't find the container with id c1c9a567c24093c8948f7b651e7e60045bb101b44f13f9d81a1a0e88453b42d9 Aug 04 18:21:02.094959 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:02.094921 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf"] Aug 04 18:21:02.097538 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:21:02.097509 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426116a3_0958_4362_89e3_b9900c902faa.slice/crio-23f5548ac9305024817c46764e0762dca9317ecc9bda96fd30cce91575097af9 WatchSource:0}: Error finding container 23f5548ac9305024817c46764e0762dca9317ecc9bda96fd30cce91575097af9: Status 404 returned error can't find the container with id 23f5548ac9305024817c46764e0762dca9317ecc9bda96fd30cce91575097af9 Aug 04 18:21:02.666389 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:02.666347 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-c9kjx" event={"ID":"3346ba4e-c7fa-4d34-83f7-2780f3b42d33","Type":"ContainerStarted","Data":"c1c9a567c24093c8948f7b651e7e60045bb101b44f13f9d81a1a0e88453b42d9"} Aug 04 18:21:02.667636 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:02.667596 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf" event={"ID":"426116a3-0958-4362-89e3-b9900c902faa","Type":"ContainerStarted","Data":"23f5548ac9305024817c46764e0762dca9317ecc9bda96fd30cce91575097af9"} Aug 04 18:21:03.685994 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:03.684806 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" event={"ID":"d45be4bd-bec4-438e-a844-4b8df8730422","Type":"ContainerStarted","Data":"1abc10c4daae5a3e67c12da75aa1c2a9b4c733bf7b9e81da7f839cde16106092"} Aug 04 18:21:03.685994 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:03.685105 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" Aug 04 18:21:03.708225 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:03.708160 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" podStartSLOduration=1.564140835 podStartE2EDuration="3.708137434s" totalImagesPullingTime="2.143996599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:21:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:21:03.706915673 +0000 UTC m=+620.958198221" watchObservedRunningTime="2026-08-04 18:21:03.708137434 +0000 UTC m=+620.959419985" Aug 04 18:21:04.692651 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:04.692540 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-c9kjx" event={"ID":"3346ba4e-c7fa-4d34-83f7-2780f3b42d33","Type":"ContainerStarted","Data":"bcdc3cf3d86eedcf406cbdb3ab151ef627afe2b15bcefebfb04f37830ce2b4ad"} Aug 04 18:21:04.693248 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:04.693136 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-c9kjx" Aug 04 18:21:04.694689 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:04.694663 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf" event={"ID":"426116a3-0958-4362-89e3-b9900c902faa","Type":"ContainerStarted","Data":"c27fafefc38d4a098d7988ceb93fc12b1367ff412c8d24a9f2a6ccb1f6b0f8ac"} Aug 04 18:21:04.694805 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:04.694707 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf" Aug 04 18:21:04.713820 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:04.713776 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-c9kjx" podStartSLOduration=1.2640219080000001 podStartE2EDuration="3.713761234s" totalImagesPullingTime="2.449739326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:21:04.712225043 +0000 UTC m=+621.963507590" watchObservedRunningTime="2026-08-04 18:21:04.713761234 +0000 UTC m=+621.965043801" Aug 04 18:21:04.734279 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:04.734237 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf" podStartSLOduration=1.3352255849999999 podStartE2EDuration="3.73422044s" totalImagesPullingTime="2.398994855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:21:04.731160978 +0000 UTC m=+621.982443524" watchObservedRunningTime="2026-08-04 18:21:04.73422044 +0000 UTC m=+621.985502986" Aug 04 18:21:09.700878 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:09.700846 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-846c96f86f-fs6jb" Aug 04 18:21:10.766770 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:10.766731 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:10.865528 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:10.865499 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:11.302111 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:11.302074 2526 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6vpxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Aug 04 18:21:11.302266 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:11.302128 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" podUID="39fbabce-fc4c-413a-ae9d-4672ba61e1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 04 18:21:11.728892 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:11.728865 2526 generic.go:356] "Generic (PLEG): container finished" podID="39fbabce-fc4c-413a-ae9d-4672ba61e1d6" containerID="4603fbc6ddc64aa45aad9a959533f76cc361d11eff9b9c953d0a557b49132f9e" exitCode=0 Aug 04 18:21:11.729103 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:11.728892 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" event={"ID":"39fbabce-fc4c-413a-ae9d-4672ba61e1d6","Type":"ContainerDied","Data":"4603fbc6ddc64aa45aad9a959533f76cc361d11eff9b9c953d0a557b49132f9e"} Aug 04 18:21:11.729375 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:11.729358 2526 scope.go:117] "RemoveContainer" containerID="4603fbc6ddc64aa45aad9a959533f76cc361d11eff9b9c953d0a557b49132f9e" Aug 04 18:21:11.730347 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:11.730318 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:21:11.730449 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:11.730383 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:21:11.730449 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:11.730419 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:21:12.735843 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:12.735807 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" event={"ID":"39fbabce-fc4c-413a-ae9d-4672ba61e1d6","Type":"ContainerStarted","Data":"ef430c2463dc690786777e4e265615932a1b14ba7ba482dc1c3cd474e245faa3"} Aug 04 18:21:12.758587 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:12.758546 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:21:12.758926 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:12.758898 2526 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6vpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Aug 04 18:21:12.759034 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:12.758954 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" podUID="39fbabce-fc4c-413a-ae9d-4672ba61e1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Aug 04 18:21:13.759341 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:13.759305 2526 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6vpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Aug 04 18:21:13.759822 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:13.759368 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" podUID="39fbabce-fc4c-413a-ae9d-4672ba61e1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Aug 04 18:21:14.758677 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:14.758633 2526 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6vpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Aug 04 18:21:14.758882 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:14.758699 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" podUID="39fbabce-fc4c-413a-ae9d-4672ba61e1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Aug 04 18:21:15.702943 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:15.702914 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zftxf" Aug 04 18:21:15.759692 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:15.759294 2526 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6vpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Aug 04 18:21:15.759692 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:15.759353 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" podUID="39fbabce-fc4c-413a-ae9d-4672ba61e1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Aug 04 18:21:16.706572 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:16.706546 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-c9kjx" Aug 04 18:21:16.768987 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:16.768502 2526 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6vpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Aug 04 18:21:16.768987 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:16.768558 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" podUID="39fbabce-fc4c-413a-ae9d-4672ba61e1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Aug 04 18:21:17.765792 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:17.765760 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:21:17.766294 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:17.766275 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:21:17.766902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:17.766888 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vpxf" Aug 04 18:21:20.653947 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:20.653911 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:34.405947 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.405911 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt"] Aug 04 18:21:34.411170 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.411149 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.413882 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.413862 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 18:21:34.414877 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.414857 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Aug 04 18:21:34.414998 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.414899 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 18:21:34.415163 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.415144 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6vn8c\"" Aug 04 18:21:34.420429 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.420187 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt"] Aug 04 18:21:34.452268 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.452234 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:34.502521 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.502491 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b52ef147-aa43-4597-9a81-e080819991b3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.502643 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.502530 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b52ef147-aa43-4597-9a81-e080819991b3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.502643 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.502555 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b52ef147-aa43-4597-9a81-e080819991b3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.502771 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.502673 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b52ef147-aa43-4597-9a81-e080819991b3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.502771 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.502707 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b52ef147-aa43-4597-9a81-e080819991b3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.502865 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.502789 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v929c\" (UniqueName: \"kubernetes.io/projected/b52ef147-aa43-4597-9a81-e080819991b3-kube-api-access-v929c\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.604153 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.604120 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v929c\" (UniqueName: \"kubernetes.io/projected/b52ef147-aa43-4597-9a81-e080819991b3-kube-api-access-v929c\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.604325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.604159 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b52ef147-aa43-4597-9a81-e080819991b3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.604325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.604201 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b52ef147-aa43-4597-9a81-e080819991b3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.604325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.604222 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b52ef147-aa43-4597-9a81-e080819991b3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.604325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.604265 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b52ef147-aa43-4597-9a81-e080819991b3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.604533 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.604402 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b52ef147-aa43-4597-9a81-e080819991b3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.604632 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.604611 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b52ef147-aa43-4597-9a81-e080819991b3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.604673 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.604648 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b52ef147-aa43-4597-9a81-e080819991b3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.604716 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.604701 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b52ef147-aa43-4597-9a81-e080819991b3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.606505 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.606479 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b52ef147-aa43-4597-9a81-e080819991b3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.606745 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.606728 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b52ef147-aa43-4597-9a81-e080819991b3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.613904 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.613882 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v929c\" (UniqueName: \"kubernetes.io/projected/b52ef147-aa43-4597-9a81-e080819991b3-kube-api-access-v929c\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jz7rt\" (UID: \"b52ef147-aa43-4597-9a81-e080819991b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.724664 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.724640 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:34.856883 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.856863 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt"] Aug 04 18:21:34.859169 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:21:34.859140 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52ef147_aa43_4597_9a81_e080819991b3.slice/crio-1ed58d5f0c4109aad1173dd9955cce8a1e4a91082881c9902b7bea500fdbe1c4 WatchSource:0}: Error finding container 1ed58d5f0c4109aad1173dd9955cce8a1e4a91082881c9902b7bea500fdbe1c4: Status 404 returned error can't find the container with id 1ed58d5f0c4109aad1173dd9955cce8a1e4a91082881c9902b7bea500fdbe1c4 Aug 04 18:21:34.860528 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:34.860507 2526 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:21:35.858780 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:35.858750 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" event={"ID":"b52ef147-aa43-4597-9a81-e080819991b3","Type":"ContainerStarted","Data":"1ed58d5f0c4109aad1173dd9955cce8a1e4a91082881c9902b7bea500fdbe1c4"} Aug 04 18:21:36.864762 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:36.864727 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" event={"ID":"b52ef147-aa43-4597-9a81-e080819991b3","Type":"ContainerStarted","Data":"a86ef350aaede7c683af85267dbff0f5e44d23297f1ed3bd4d030995e2830eb1"} Aug 04 18:21:36.883651 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:36.883609 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" podStartSLOduration=1.83639034 podStartE2EDuration="2.883595612s" totalImagesPullingTime="1.047205272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:21:36.88223668 +0000 UTC m=+654.133519227" watchObservedRunningTime="2026-08-04 18:21:36.883595612 +0000 UTC m=+654.134878175" Aug 04 18:21:44.725014 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:44.724939 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:44.725467 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:44.725024 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:44.737547 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:44.737527 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:44.911208 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:44.911182 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jz7rt" Aug 04 18:21:47.672834 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:47.672801 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:52.004263 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.004222 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr"] Aug 04 18:21:52.008290 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.008266 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.010990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.010956 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Aug 04 18:21:52.018918 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.018891 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr"] Aug 04 18:21:52.041273 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.041244 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:52.147536 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.147495 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c6e331a-0343-4b1f-b95d-800ceba00cf7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.147705 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.147542 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c6e331a-0343-4b1f-b95d-800ceba00cf7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.147705 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.147576 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c6e331a-0343-4b1f-b95d-800ceba00cf7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.147705 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.147597 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c6e331a-0343-4b1f-b95d-800ceba00cf7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.147705 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.147689 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c6e331a-0343-4b1f-b95d-800ceba00cf7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.147867 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.147741 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzh8q\" (UniqueName: \"kubernetes.io/projected/2c6e331a-0343-4b1f-b95d-800ceba00cf7-kube-api-access-qzh8q\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.248880 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.248839 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c6e331a-0343-4b1f-b95d-800ceba00cf7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.248880 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.248881 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c6e331a-0343-4b1f-b95d-800ceba00cf7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.249151 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.248904 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c6e331a-0343-4b1f-b95d-800ceba00cf7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.249151 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.248924 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c6e331a-0343-4b1f-b95d-800ceba00cf7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.249151 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.248960 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c6e331a-0343-4b1f-b95d-800ceba00cf7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.249151 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.249011 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzh8q\" (UniqueName: \"kubernetes.io/projected/2c6e331a-0343-4b1f-b95d-800ceba00cf7-kube-api-access-qzh8q\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.249355 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.249330 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c6e331a-0343-4b1f-b95d-800ceba00cf7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.249466 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.249416 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c6e331a-0343-4b1f-b95d-800ceba00cf7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.249466 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.249449 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c6e331a-0343-4b1f-b95d-800ceba00cf7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.251326 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.251292 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c6e331a-0343-4b1f-b95d-800ceba00cf7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.251511 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.251495 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c6e331a-0343-4b1f-b95d-800ceba00cf7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.259635 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.259583 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzh8q\" (UniqueName: \"kubernetes.io/projected/2c6e331a-0343-4b1f-b95d-800ceba00cf7-kube-api-access-qzh8q\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wg5tr\" (UID: \"2c6e331a-0343-4b1f-b95d-800ceba00cf7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.320302 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.320270 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:21:52.476678 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.476649 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr"] Aug 04 18:21:52.478724 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:21:52.478691 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6e331a_0343_4b1f_b95d_800ceba00cf7.slice/crio-2a2023ee5552b0a8e0fa2217db183aaabbc43333b592675b08ad00f22fa97d27 WatchSource:0}: Error finding container 2a2023ee5552b0a8e0fa2217db183aaabbc43333b592675b08ad00f22fa97d27: Status 404 returned error can't find the container with id 2a2023ee5552b0a8e0fa2217db183aaabbc43333b592675b08ad00f22fa97d27 Aug 04 18:21:52.941733 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.941641 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" event={"ID":"2c6e331a-0343-4b1f-b95d-800ceba00cf7","Type":"ContainerStarted","Data":"d8d8e3652ebec0f4b6606cb10c96d87e5a77e435b505c64f9abbc8b80c0097f9"} Aug 04 18:21:52.941733 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.941707 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" event={"ID":"2c6e331a-0343-4b1f-b95d-800ceba00cf7","Type":"ContainerStarted","Data":"2a2023ee5552b0a8e0fa2217db183aaabbc43333b592675b08ad00f22fa97d27"} Aug 04 18:21:52.964542 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:21:52.964495 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" podStartSLOduration=1.791189606 podStartE2EDuration="1.964479805s" totalImagesPullingTime="173.290199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:21:52.962360736 +0000 UTC m=+670.213643283" watchObservedRunningTime="2026-08-04 18:21:52.964479805 +0000 UTC m=+670.215762351" Aug 04 18:22:00.389985 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:00.389942 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:02.320846 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:02.320807 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:22:02.321310 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:02.320887 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:22:02.333080 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:02.333061 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:22:03.002708 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:03.002679 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wg5tr" Aug 04 18:22:10.672798 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:10.672759 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:12.628278 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.628231 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks"] Aug 04 18:22:12.633299 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.633269 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.636010 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.635987 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Aug 04 18:22:12.652194 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.652170 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks"] Aug 04 18:22:12.684372 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.684345 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:12.727165 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.727142 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.727288 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.727172 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.727288 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.727192 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.727288 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.727216 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.727392 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.727316 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.727392 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.727362 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krlbz\" (UniqueName: \"kubernetes.io/projected/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-kube-api-access-krlbz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.828278 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.828248 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.828278 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.828285 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.828510 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.828415 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.828568 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.828502 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krlbz\" (UniqueName: \"kubernetes.io/projected/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-kube-api-access-krlbz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.828626 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.828606 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.828626 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.828616 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.828730 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.828636 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.828807 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.828782 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.828961 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.828944 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.830513 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.830485 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.830827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.830810 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.838350 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.838326 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krlbz\" (UniqueName: \"kubernetes.io/projected/a8d9ecd5-91f5-4114-9bb9-40afc20e9d59-kube-api-access-krlbz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks\" (UID: \"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:12.942638 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:12.942544 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:13.085631 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:13.085591 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks"] Aug 04 18:22:13.087781 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:22:13.087753 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8d9ecd5_91f5_4114_9bb9_40afc20e9d59.slice/crio-696720de5da5adf93ac0088982fd42ad0f8e2be725eb2854d6c9208749035850 WatchSource:0}: Error finding container 696720de5da5adf93ac0088982fd42ad0f8e2be725eb2854d6c9208749035850: Status 404 returned error can't find the container with id 696720de5da5adf93ac0088982fd42ad0f8e2be725eb2854d6c9208749035850 Aug 04 18:22:14.040855 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:14.040819 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" event={"ID":"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59","Type":"ContainerStarted","Data":"a54ef16cd2d87b73f96596151b415c49981f4239cc864dce227074a8a8ac4ee2"} Aug 04 18:22:14.041250 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:14.040870 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" event={"ID":"a8d9ecd5-91f5-4114-9bb9-40afc20e9d59","Type":"ContainerStarted","Data":"696720de5da5adf93ac0088982fd42ad0f8e2be725eb2854d6c9208749035850"} Aug 04 18:22:14.061876 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:14.061833 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" podStartSLOduration=1.86569878 podStartE2EDuration="2.061818307s" totalImagesPullingTime="196.119527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:22:14.06022175 +0000 UTC m=+691.311504288" watchObservedRunningTime="2026-08-04 18:22:14.061818307 +0000 UTC m=+691.313100853" Aug 04 18:22:22.943670 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:22.943627 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:22.943670 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:22.943679 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:22.955888 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:22.955867 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:23.093256 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:23.093225 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks" Aug 04 18:22:57.134227 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:57.134142 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:57.233799 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:57.233765 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:58.376460 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:58.376423 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:58.480620 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:22:58.480583 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:27.965723 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:27.965683 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:28.032357 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:28.032327 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:47.543907 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:47.543868 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cffffc5dc-2zdrb"] Aug 04 18:23:47.544403 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:47.544129 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" podUID="ed612c71-3187-43f9-bcea-bdfd39590463" containerName="manager" containerID="cri-o://b2aa3647cc5637f9f2d917e4463aec2b61e6101e3a2707d99865dd120a0c12f2" gracePeriod=30 Aug 04 18:23:47.793568 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:47.793543 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" Aug 04 18:23:47.919076 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:47.919004 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed612c71-3187-43f9-bcea-bdfd39590463-webhook-cert\") pod \"ed612c71-3187-43f9-bcea-bdfd39590463\" (UID: \"ed612c71-3187-43f9-bcea-bdfd39590463\") " Aug 04 18:23:47.919076 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:47.919045 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9zmm\" (UniqueName: \"kubernetes.io/projected/ed612c71-3187-43f9-bcea-bdfd39590463-kube-api-access-b9zmm\") pod \"ed612c71-3187-43f9-bcea-bdfd39590463\" (UID: \"ed612c71-3187-43f9-bcea-bdfd39590463\") " Aug 04 18:23:47.920939 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:47.920914 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed612c71-3187-43f9-bcea-bdfd39590463-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ed612c71-3187-43f9-bcea-bdfd39590463" (UID: "ed612c71-3187-43f9-bcea-bdfd39590463"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:23:47.921077 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:47.921056 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed612c71-3187-43f9-bcea-bdfd39590463-kube-api-access-b9zmm" (OuterVolumeSpecName: "kube-api-access-b9zmm") pod "ed612c71-3187-43f9-bcea-bdfd39590463" (UID: "ed612c71-3187-43f9-bcea-bdfd39590463"). InnerVolumeSpecName "kube-api-access-b9zmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:23:48.020421 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:48.020396 2526 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed612c71-3187-43f9-bcea-bdfd39590463-webhook-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:23:48.020421 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:48.020420 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9zmm\" (UniqueName: \"kubernetes.io/projected/ed612c71-3187-43f9-bcea-bdfd39590463-kube-api-access-b9zmm\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:23:48.472933 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:48.472898 2526 generic.go:356] "Generic (PLEG): container finished" podID="ed612c71-3187-43f9-bcea-bdfd39590463" containerID="b2aa3647cc5637f9f2d917e4463aec2b61e6101e3a2707d99865dd120a0c12f2" exitCode=0 Aug 04 18:23:48.473098 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:48.472942 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" Aug 04 18:23:48.473098 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:48.472990 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" event={"ID":"ed612c71-3187-43f9-bcea-bdfd39590463","Type":"ContainerDied","Data":"b2aa3647cc5637f9f2d917e4463aec2b61e6101e3a2707d99865dd120a0c12f2"} Aug 04 18:23:48.473098 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:48.473041 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cffffc5dc-2zdrb" event={"ID":"ed612c71-3187-43f9-bcea-bdfd39590463","Type":"ContainerDied","Data":"de1d5bcab2599821cb032ae32ba3b3d6de23b86f1b767dd6e8a30bd2c86155ff"} Aug 04 18:23:48.473098 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:48.473063 2526 scope.go:117] "RemoveContainer" containerID="b2aa3647cc5637f9f2d917e4463aec2b61e6101e3a2707d99865dd120a0c12f2" Aug 04 18:23:48.482106 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:48.482090 2526 scope.go:117] "RemoveContainer" containerID="b2aa3647cc5637f9f2d917e4463aec2b61e6101e3a2707d99865dd120a0c12f2" Aug 04 18:23:48.482373 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:23:48.482356 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2aa3647cc5637f9f2d917e4463aec2b61e6101e3a2707d99865dd120a0c12f2\": container with ID starting with b2aa3647cc5637f9f2d917e4463aec2b61e6101e3a2707d99865dd120a0c12f2 not found: ID does not exist" containerID="b2aa3647cc5637f9f2d917e4463aec2b61e6101e3a2707d99865dd120a0c12f2" Aug 04 18:23:48.482413 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:48.482380 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2aa3647cc5637f9f2d917e4463aec2b61e6101e3a2707d99865dd120a0c12f2"} err="failed to get container status \"b2aa3647cc5637f9f2d917e4463aec2b61e6101e3a2707d99865dd120a0c12f2\": rpc error: code = NotFound desc = could not find container \"b2aa3647cc5637f9f2d917e4463aec2b61e6101e3a2707d99865dd120a0c12f2\": container with ID starting with b2aa3647cc5637f9f2d917e4463aec2b61e6101e3a2707d99865dd120a0c12f2 not found: ID does not exist" Aug 04 18:23:48.503189 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:48.503168 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cffffc5dc-2zdrb"] Aug 04 18:23:48.510729 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:48.510710 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cffffc5dc-2zdrb"] Aug 04 18:23:49.182529 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:23:49.182495 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed612c71-3187-43f9-bcea-bdfd39590463" path="/var/lib/kubelet/pods/ed612c71-3187-43f9-bcea-bdfd39590463/volumes" Aug 04 18:24:17.010759 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:17.010673 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:49.430762 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.430734 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cffffc5dc-zrzt5"] Aug 04 18:24:49.431234 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.431104 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed612c71-3187-43f9-bcea-bdfd39590463" containerName="manager" Aug 04 18:24:49.431234 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.431116 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed612c71-3187-43f9-bcea-bdfd39590463" containerName="manager" Aug 04 18:24:49.431234 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.431181 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed612c71-3187-43f9-bcea-bdfd39590463" containerName="manager" Aug 04 18:24:49.435323 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.435301 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" Aug 04 18:24:49.438932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.438910 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j4d7j\"" Aug 04 18:24:49.439049 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.438910 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 18:24:49.439049 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.438945 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 18:24:49.446363 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.446345 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cffffc5dc-zrzt5"] Aug 04 18:24:49.506908 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.506884 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7pz9\" (UniqueName: \"kubernetes.io/projected/c1a7e28a-4438-4239-97e3-18281259d181-kube-api-access-g7pz9\") pod \"maas-controller-5cffffc5dc-zrzt5\" (UID: \"c1a7e28a-4438-4239-97e3-18281259d181\") " pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" Aug 04 18:24:49.507046 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.506925 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1a7e28a-4438-4239-97e3-18281259d181-webhook-cert\") pod \"maas-controller-5cffffc5dc-zrzt5\" (UID: \"c1a7e28a-4438-4239-97e3-18281259d181\") " pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" Aug 04 18:24:49.607388 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.607364 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7pz9\" (UniqueName: \"kubernetes.io/projected/c1a7e28a-4438-4239-97e3-18281259d181-kube-api-access-g7pz9\") pod \"maas-controller-5cffffc5dc-zrzt5\" (UID: \"c1a7e28a-4438-4239-97e3-18281259d181\") " pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" Aug 04 18:24:49.607489 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.607404 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1a7e28a-4438-4239-97e3-18281259d181-webhook-cert\") pod \"maas-controller-5cffffc5dc-zrzt5\" (UID: \"c1a7e28a-4438-4239-97e3-18281259d181\") " pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" Aug 04 18:24:49.609770 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.609749 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1a7e28a-4438-4239-97e3-18281259d181-webhook-cert\") pod \"maas-controller-5cffffc5dc-zrzt5\" (UID: \"c1a7e28a-4438-4239-97e3-18281259d181\") " pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" Aug 04 18:24:49.618811 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.618792 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7pz9\" (UniqueName: \"kubernetes.io/projected/c1a7e28a-4438-4239-97e3-18281259d181-kube-api-access-g7pz9\") pod \"maas-controller-5cffffc5dc-zrzt5\" (UID: \"c1a7e28a-4438-4239-97e3-18281259d181\") " pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" Aug 04 18:24:49.745351 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:49.745330 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" Aug 04 18:24:50.102346 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:50.102319 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cffffc5dc-zrzt5"] Aug 04 18:24:50.103557 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:24:50.103529 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a7e28a_4438_4239_97e3_18281259d181.slice/crio-43bfd4f3f508e0c7bf59f1962b5c72f68e04b38532e3cee3614e5b820adcc931 WatchSource:0}: Error finding container 43bfd4f3f508e0c7bf59f1962b5c72f68e04b38532e3cee3614e5b820adcc931: Status 404 returned error can't find the container with id 43bfd4f3f508e0c7bf59f1962b5c72f68e04b38532e3cee3614e5b820adcc931 Aug 04 18:24:50.760745 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:50.760707 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" event={"ID":"c1a7e28a-4438-4239-97e3-18281259d181","Type":"ContainerStarted","Data":"07ce335db147a581a03d2898ccb5543070e36a83c8fa20f9b3fe773c09a180f5"} Aug 04 18:24:50.761120 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:50.760781 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" event={"ID":"c1a7e28a-4438-4239-97e3-18281259d181","Type":"ContainerStarted","Data":"43bfd4f3f508e0c7bf59f1962b5c72f68e04b38532e3cee3614e5b820adcc931"} Aug 04 18:24:50.761120 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:50.760862 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" Aug 04 18:24:50.781115 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:24:50.781072 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" podStartSLOduration=1.3253345890000001 podStartE2EDuration="1.781059337s" totalImagesPullingTime="455.724748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:24:50.77968617 +0000 UTC m=+848.030968719" watchObservedRunningTime="2026-08-04 18:24:50.781059337 +0000 UTC m=+848.032341922" Aug 04 18:25:01.768526 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:01.768493 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" Aug 04 18:25:06.354943 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.354911 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:25:06.454450 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.454421 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cffffc5dc-zrzt5"] Aug 04 18:25:06.454645 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.454626 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" podUID="c1a7e28a-4438-4239-97e3-18281259d181" containerName="manager" containerID="cri-o://07ce335db147a581a03d2898ccb5543070e36a83c8fa20f9b3fe773c09a180f5" gracePeriod=30 Aug 04 18:25:06.693761 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.693737 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" Aug 04 18:25:06.765938 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.765911 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1a7e28a-4438-4239-97e3-18281259d181-webhook-cert\") pod \"c1a7e28a-4438-4239-97e3-18281259d181\" (UID: \"c1a7e28a-4438-4239-97e3-18281259d181\") " Aug 04 18:25:06.766130 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.765949 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7pz9\" (UniqueName: \"kubernetes.io/projected/c1a7e28a-4438-4239-97e3-18281259d181-kube-api-access-g7pz9\") pod \"c1a7e28a-4438-4239-97e3-18281259d181\" (UID: \"c1a7e28a-4438-4239-97e3-18281259d181\") " Aug 04 18:25:06.767933 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.767903 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a7e28a-4438-4239-97e3-18281259d181-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c1a7e28a-4438-4239-97e3-18281259d181" (UID: "c1a7e28a-4438-4239-97e3-18281259d181"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:25:06.768058 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.767986 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a7e28a-4438-4239-97e3-18281259d181-kube-api-access-g7pz9" (OuterVolumeSpecName: "kube-api-access-g7pz9") pod "c1a7e28a-4438-4239-97e3-18281259d181" (UID: "c1a7e28a-4438-4239-97e3-18281259d181"). InnerVolumeSpecName "kube-api-access-g7pz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:25:06.838236 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.838198 2526 generic.go:356] "Generic (PLEG): container finished" podID="c1a7e28a-4438-4239-97e3-18281259d181" containerID="07ce335db147a581a03d2898ccb5543070e36a83c8fa20f9b3fe773c09a180f5" exitCode=0 Aug 04 18:25:06.838420 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.838241 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" Aug 04 18:25:06.838420 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.838242 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" event={"ID":"c1a7e28a-4438-4239-97e3-18281259d181","Type":"ContainerDied","Data":"07ce335db147a581a03d2898ccb5543070e36a83c8fa20f9b3fe773c09a180f5"} Aug 04 18:25:06.838420 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.838363 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cffffc5dc-zrzt5" event={"ID":"c1a7e28a-4438-4239-97e3-18281259d181","Type":"ContainerDied","Data":"43bfd4f3f508e0c7bf59f1962b5c72f68e04b38532e3cee3614e5b820adcc931"} Aug 04 18:25:06.838420 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.838391 2526 scope.go:117] "RemoveContainer" containerID="07ce335db147a581a03d2898ccb5543070e36a83c8fa20f9b3fe773c09a180f5" Aug 04 18:25:06.846946 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.846933 2526 scope.go:117] "RemoveContainer" containerID="07ce335db147a581a03d2898ccb5543070e36a83c8fa20f9b3fe773c09a180f5" Aug 04 18:25:06.847211 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:25:06.847191 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ce335db147a581a03d2898ccb5543070e36a83c8fa20f9b3fe773c09a180f5\": container with ID starting with 07ce335db147a581a03d2898ccb5543070e36a83c8fa20f9b3fe773c09a180f5 not found: ID does not exist" containerID="07ce335db147a581a03d2898ccb5543070e36a83c8fa20f9b3fe773c09a180f5" Aug 04 18:25:06.847261 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.847218 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ce335db147a581a03d2898ccb5543070e36a83c8fa20f9b3fe773c09a180f5"} err="failed to get container status \"07ce335db147a581a03d2898ccb5543070e36a83c8fa20f9b3fe773c09a180f5\": rpc error: code = NotFound desc = could not find container \"07ce335db147a581a03d2898ccb5543070e36a83c8fa20f9b3fe773c09a180f5\": container with ID starting with 07ce335db147a581a03d2898ccb5543070e36a83c8fa20f9b3fe773c09a180f5 not found: ID does not exist" Aug 04 18:25:06.863105 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.863084 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cffffc5dc-zrzt5"] Aug 04 18:25:06.866627 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.866603 2526 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1a7e28a-4438-4239-97e3-18281259d181-webhook-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:25:06.866738 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.866631 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7pz9\" (UniqueName: \"kubernetes.io/projected/c1a7e28a-4438-4239-97e3-18281259d181-kube-api-access-g7pz9\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:25:06.868305 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:06.868285 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cffffc5dc-zrzt5"] Aug 04 18:25:07.180980 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.180944 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a7e28a-4438-4239-97e3-18281259d181" path="/var/lib/kubelet/pods/c1a7e28a-4438-4239-97e3-18281259d181/volumes" Aug 04 18:25:07.610700 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.610667 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:25:07.730960 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.730931 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cffffc5dc-blwt8"] Aug 04 18:25:07.731326 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.731314 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a7e28a-4438-4239-97e3-18281259d181" containerName="manager" Aug 04 18:25:07.731373 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.731328 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a7e28a-4438-4239-97e3-18281259d181" containerName="manager" Aug 04 18:25:07.731411 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.731394 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a7e28a-4438-4239-97e3-18281259d181" containerName="manager" Aug 04 18:25:07.735607 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.735583 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cffffc5dc-blwt8" Aug 04 18:25:07.738853 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.738835 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 18:25:07.738853 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.738844 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j4d7j\"" Aug 04 18:25:07.739034 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.738835 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 18:25:07.744328 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.744305 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cffffc5dc-blwt8"] Aug 04 18:25:07.774318 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.774294 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7e54ecf-6cca-42e8-a708-edc70403685f-webhook-cert\") pod \"maas-controller-5cffffc5dc-blwt8\" (UID: \"a7e54ecf-6cca-42e8-a708-edc70403685f\") " pod="opendatahub/maas-controller-5cffffc5dc-blwt8" Aug 04 18:25:07.774409 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.774329 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctvg7\" (UniqueName: \"kubernetes.io/projected/a7e54ecf-6cca-42e8-a708-edc70403685f-kube-api-access-ctvg7\") pod \"maas-controller-5cffffc5dc-blwt8\" (UID: \"a7e54ecf-6cca-42e8-a708-edc70403685f\") " pod="opendatahub/maas-controller-5cffffc5dc-blwt8" Aug 04 18:25:07.874866 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.874795 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7e54ecf-6cca-42e8-a708-edc70403685f-webhook-cert\") pod \"maas-controller-5cffffc5dc-blwt8\" (UID: \"a7e54ecf-6cca-42e8-a708-edc70403685f\") " pod="opendatahub/maas-controller-5cffffc5dc-blwt8" Aug 04 18:25:07.874866 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.874841 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctvg7\" (UniqueName: \"kubernetes.io/projected/a7e54ecf-6cca-42e8-a708-edc70403685f-kube-api-access-ctvg7\") pod \"maas-controller-5cffffc5dc-blwt8\" (UID: \"a7e54ecf-6cca-42e8-a708-edc70403685f\") " pod="opendatahub/maas-controller-5cffffc5dc-blwt8" Aug 04 18:25:07.877192 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.877172 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7e54ecf-6cca-42e8-a708-edc70403685f-webhook-cert\") pod \"maas-controller-5cffffc5dc-blwt8\" (UID: \"a7e54ecf-6cca-42e8-a708-edc70403685f\") " pod="opendatahub/maas-controller-5cffffc5dc-blwt8" Aug 04 18:25:07.884982 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:07.884947 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctvg7\" (UniqueName: \"kubernetes.io/projected/a7e54ecf-6cca-42e8-a708-edc70403685f-kube-api-access-ctvg7\") pod \"maas-controller-5cffffc5dc-blwt8\" (UID: \"a7e54ecf-6cca-42e8-a708-edc70403685f\") " pod="opendatahub/maas-controller-5cffffc5dc-blwt8" Aug 04 18:25:08.047245 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:08.047215 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cffffc5dc-blwt8" Aug 04 18:25:08.180315 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:08.180292 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cffffc5dc-blwt8"] Aug 04 18:25:08.181619 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:25:08.181592 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e54ecf_6cca_42e8_a708_edc70403685f.slice/crio-0cf5e90f8bf015901836170ff4a8f7384c33e975baded614bdf5e9e7838954ae WatchSource:0}: Error finding container 0cf5e90f8bf015901836170ff4a8f7384c33e975baded614bdf5e9e7838954ae: Status 404 returned error can't find the container with id 0cf5e90f8bf015901836170ff4a8f7384c33e975baded614bdf5e9e7838954ae Aug 04 18:25:08.854417 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:08.854362 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cffffc5dc-blwt8" event={"ID":"a7e54ecf-6cca-42e8-a708-edc70403685f","Type":"ContainerStarted","Data":"395a2a9e98d0670cafa1a8cda982e3aa43a69964d6f777491d1133cd015cabd2"} Aug 04 18:25:08.854864 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:08.854467 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cffffc5dc-blwt8" Aug 04 18:25:08.854864 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:08.854490 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cffffc5dc-blwt8" event={"ID":"a7e54ecf-6cca-42e8-a708-edc70403685f","Type":"ContainerStarted","Data":"0cf5e90f8bf015901836170ff4a8f7384c33e975baded614bdf5e9e7838954ae"} Aug 04 18:25:08.876039 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:08.875993 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cffffc5dc-blwt8" podStartSLOduration=1.347505467 podStartE2EDuration="1.875978531s" totalImagesPullingTime="528.473064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:25:08.874178184 +0000 UTC m=+866.125460731" watchObservedRunningTime="2026-08-04 18:25:08.875978531 +0000 UTC m=+866.127261070" Aug 04 18:25:19.861900 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:25:19.861868 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cffffc5dc-blwt8" Aug 04 18:26:13.481135 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:26:13.481096 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:13.687494 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:26:13.687455 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:13.793306 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:26:13.793229 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:20.678026 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:26:20.677988 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:20.771315 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:26:20.771264 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:20.921241 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:26:20.921209 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:21.034115 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:26:21.034084 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:21.181900 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:26:21.181868 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:21.441892 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:26:21.441801 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:29.117583 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:26:29.117544 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:31.022313 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:26:31.022271 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:29.587793 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:27:29.587756 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:29.666139 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:27:29.666102 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:40.743439 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:27:40.743397 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:40.885143 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:27:40.885091 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:41.035163 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:27:41.035056 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:29:40.722101 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:29:40.722068 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:30:05.617234 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:30:05.617151 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:30:05.700404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:30:05.700359 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:30:09.242065 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:30:09.242020 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:35:42.058765 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:35:42.058730 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:35:44.009341 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:35:44.009304 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:23.366748 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:36:23.366670 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:24.441473 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:36:24.441441 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:24.909632 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:36:24.909600 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:18.688205 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:38:18.688172 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:39:48.850296 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:39:48.850266 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:39:59.376646 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:39:59.376615 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:40:25.851486 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:25.851448 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:40:29.148502 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:29.148461 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:40:29.283488 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:29.283448 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:40:29.736949 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:29.736919 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:40:36.852999 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:36.852902 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb"] Aug 04 18:40:36.856511 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:36.856493 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:36.859130 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:36.859103 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9756c1e3-openshift-default-dockercfg-6r6mr\"" Aug 04 18:40:36.887618 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:36.887593 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb"] Aug 04 18:40:36.949715 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:36.949678 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkpht\" (UniqueName: \"kubernetes.io/projected/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-kube-api-access-nkpht\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:36.949897 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:36.949724 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-credential-socket\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:36.949897 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:36.949773 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istiod-ca-cert\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:36.949897 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:36.949814 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-envoy\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:36.949897 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:36.949830 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-token\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:36.949897 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:36.949846 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-data\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:36.950127 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:36.949907 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-workload-socket\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:36.950127 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:36.949934 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-workload-certs\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:36.950127 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:36.949973 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-podinfo\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.050829 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.050798 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-workload-socket\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.051008 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.050835 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-workload-certs\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.051008 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.050854 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-podinfo\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.051008 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.050896 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkpht\" (UniqueName: \"kubernetes.io/projected/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-kube-api-access-nkpht\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.051008 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.050926 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-credential-socket\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.051008 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.050997 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istiod-ca-cert\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.051294 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.051048 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-envoy\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.051294 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.051070 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-token\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.051294 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.051095 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-data\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.051294 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.051242 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-workload-socket\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.051481 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.051434 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-workload-certs\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.051594 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.051568 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-credential-socket\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.051594 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.051587 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-data\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.051800 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.051778 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istiod-ca-cert\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.053544 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.053522 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-envoy\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.053653 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.053567 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-podinfo\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.067127 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.067103 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-token\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.067243 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.067219 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkpht\" (UniqueName: \"kubernetes.io/projected/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-kube-api-access-nkpht\") pod \"e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.117239 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.117178 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:40:37.167129 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.167106 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:37.221607 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.221540 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:40:37.310537 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.310512 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb"] Aug 04 18:40:37.313657 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:40:37.313626 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803fee5d_ae6c_4309_a1bc_6d03be17c4a8.slice/crio-0044e8171d7e2ae40c9090c186ae7fa32f93bccd735a51d7d6e091a8e85b7ecd WatchSource:0}: Error finding container 0044e8171d7e2ae40c9090c186ae7fa32f93bccd735a51d7d6e091a8e85b7ecd: Status 404 returned error can't find the container with id 0044e8171d7e2ae40c9090c186ae7fa32f93bccd735a51d7d6e091a8e85b7ecd Aug 04 18:40:37.315721 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.315685 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:40:37.315782 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.315753 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:40:37.315828 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:37.315781 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:40:38.249923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:38.249882 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" event={"ID":"803fee5d-ae6c-4309-a1bc-6d03be17c4a8","Type":"ContainerStarted","Data":"5ae26888c5386e123238e54ca09d83ce21efd026630bede8f9e143b1176d6074"} Aug 04 18:40:38.250324 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:38.250005 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" event={"ID":"803fee5d-ae6c-4309-a1bc-6d03be17c4a8","Type":"ContainerStarted","Data":"0044e8171d7e2ae40c9090c186ae7fa32f93bccd735a51d7d6e091a8e85b7ecd"} Aug 04 18:40:38.295730 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:38.295687 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" podStartSLOduration=2.295672576 podStartE2EDuration="2.295672576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:40:38.295279503 +0000 UTC m=+1795.546562051" watchObservedRunningTime="2026-08-04 18:40:38.295672576 +0000 UTC m=+1795.546955123" Aug 04 18:40:39.168001 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:39.167955 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:39.172810 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:39.172790 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:39.254484 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:39.254454 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:39.255353 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:39.255335 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:39.708493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:39.708452 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:40:39.814134 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:39.814103 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:40:40.824773 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:40.824748 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk"] Aug 04 18:40:40.828359 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:40.828333 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" Aug 04 18:40:40.834633 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:40.834608 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9756c1e3-serving-cert\"" Aug 04 18:40:40.852614 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:40.852537 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk"] Aug 04 18:40:40.885820 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:40.885788 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbca186-f04a-4787-9127-8e86b8021f2f-openshift-service-ca\") pod \"maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk\" (UID: \"ffbca186-f04a-4787-9127-8e86b8021f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" Aug 04 18:40:40.886008 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:40.885831 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffbca186-f04a-4787-9127-8e86b8021f2f-maas-api-tls\") pod \"maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk\" (UID: \"ffbca186-f04a-4787-9127-8e86b8021f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" Aug 04 18:40:40.886008 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:40.885899 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwdgt\" (UniqueName: \"kubernetes.io/projected/ffbca186-f04a-4787-9127-8e86b8021f2f-kube-api-access-gwdgt\") pod \"maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk\" (UID: \"ffbca186-f04a-4787-9127-8e86b8021f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" Aug 04 18:40:40.978663 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:40.978617 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:40:40.986907 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:40.986884 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbca186-f04a-4787-9127-8e86b8021f2f-openshift-service-ca\") pod \"maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk\" (UID: \"ffbca186-f04a-4787-9127-8e86b8021f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" Aug 04 18:40:40.987060 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:40.986921 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffbca186-f04a-4787-9127-8e86b8021f2f-maas-api-tls\") pod \"maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk\" (UID: \"ffbca186-f04a-4787-9127-8e86b8021f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" Aug 04 18:40:40.987060 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:40.986989 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwdgt\" (UniqueName: \"kubernetes.io/projected/ffbca186-f04a-4787-9127-8e86b8021f2f-kube-api-access-gwdgt\") pod \"maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk\" (UID: \"ffbca186-f04a-4787-9127-8e86b8021f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" Aug 04 18:40:40.987668 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:40.987647 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbca186-f04a-4787-9127-8e86b8021f2f-openshift-service-ca\") pod \"maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk\" (UID: \"ffbca186-f04a-4787-9127-8e86b8021f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" Aug 04 18:40:40.989643 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:40.989624 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffbca186-f04a-4787-9127-8e86b8021f2f-maas-api-tls\") pod \"maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk\" (UID: \"ffbca186-f04a-4787-9127-8e86b8021f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" Aug 04 18:40:41.011150 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:41.011123 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwdgt\" (UniqueName: \"kubernetes.io/projected/ffbca186-f04a-4787-9127-8e86b8021f2f-kube-api-access-gwdgt\") pod \"maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk\" (UID: \"ffbca186-f04a-4787-9127-8e86b8021f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" Aug 04 18:40:41.140664 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:41.140579 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" Aug 04 18:40:41.217113 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:41.217087 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:40:41.280568 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:41.280544 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk"] Aug 04 18:40:41.282100 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:40:41.282070 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbca186_f04a_4787_9127_8e86b8021f2f.slice/crio-c326ed28ee6b04078196b79688022e5a5152341a659916b3e45b53df295b2474 WatchSource:0}: Error finding container c326ed28ee6b04078196b79688022e5a5152341a659916b3e45b53df295b2474: Status 404 returned error can't find the container with id c326ed28ee6b04078196b79688022e5a5152341a659916b3e45b53df295b2474 Aug 04 18:40:41.283441 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:41.283422 2526 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:40:41.639408 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:41.639362 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:40:41.832394 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:41.832332 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:40:41.933000 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:41.932909 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz"] Aug 04 18:40:41.937374 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:41.937355 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" Aug 04 18:40:41.943568 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:41.943549 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9756c1e3-dockercfg-rbdv5\"" Aug 04 18:40:41.953588 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:41.953571 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9756c1e3\"" Aug 04 18:40:41.959298 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:41.959272 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz"] Aug 04 18:40:41.996044 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:41.996021 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcvcq\" (UniqueName: \"kubernetes.io/projected/0611b20f-b263-4167-a7c7-75e045e9fac4-kube-api-access-rcvcq\") pod \"payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz\" (UID: \"0611b20f-b263-4167-a7c7-75e045e9fac4\") " pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" Aug 04 18:40:41.996161 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:41.996053 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0611b20f-b263-4167-a7c7-75e045e9fac4-plugins-config-volume\") pod \"payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz\" (UID: \"0611b20f-b263-4167-a7c7-75e045e9fac4\") " pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" Aug 04 18:40:42.097503 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:42.097470 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcvcq\" (UniqueName: \"kubernetes.io/projected/0611b20f-b263-4167-a7c7-75e045e9fac4-kube-api-access-rcvcq\") pod \"payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz\" (UID: \"0611b20f-b263-4167-a7c7-75e045e9fac4\") " pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" Aug 04 18:40:42.097656 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:42.097523 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0611b20f-b263-4167-a7c7-75e045e9fac4-plugins-config-volume\") pod \"payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz\" (UID: \"0611b20f-b263-4167-a7c7-75e045e9fac4\") " pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" Aug 04 18:40:42.098432 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:42.098408 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0611b20f-b263-4167-a7c7-75e045e9fac4-plugins-config-volume\") pod \"payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz\" (UID: \"0611b20f-b263-4167-a7c7-75e045e9fac4\") " pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" Aug 04 18:40:42.111193 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:42.111163 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcvcq\" (UniqueName: \"kubernetes.io/projected/0611b20f-b263-4167-a7c7-75e045e9fac4-kube-api-access-rcvcq\") pod \"payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz\" (UID: \"0611b20f-b263-4167-a7c7-75e045e9fac4\") " pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" Aug 04 18:40:42.249109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:42.249073 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" Aug 04 18:40:42.271092 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:42.271059 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" event={"ID":"ffbca186-f04a-4787-9127-8e86b8021f2f","Type":"ContainerStarted","Data":"c326ed28ee6b04078196b79688022e5a5152341a659916b3e45b53df295b2474"} Aug 04 18:40:42.394679 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:42.394656 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz"] Aug 04 18:40:42.396554 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:40:42.396526 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0611b20f_b263_4167_a7c7_75e045e9fac4.slice/crio-23822e748d110f76a13a3880a4d4c0312f67486fa19dec68d2e98381120be2ea WatchSource:0}: Error finding container 23822e748d110f76a13a3880a4d4c0312f67486fa19dec68d2e98381120be2ea: Status 404 returned error can't find the container with id 23822e748d110f76a13a3880a4d4c0312f67486fa19dec68d2e98381120be2ea Aug 04 18:40:43.277739 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:43.277623 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" event={"ID":"ffbca186-f04a-4787-9127-8e86b8021f2f","Type":"ContainerStarted","Data":"2bf7d5191ed63de634de263ee901a36099e20d8c6d8458c17e5fce81bf24402d"} Aug 04 18:40:43.277739 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:43.277710 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" Aug 04 18:40:43.279237 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:43.279211 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" event={"ID":"0611b20f-b263-4167-a7c7-75e045e9fac4","Type":"ContainerStarted","Data":"74277d0d94c40eb09b143c3bca60cb0a8f28ae6b70c93a34339c4e6b20f2e8e2"} Aug 04 18:40:43.279374 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:43.279258 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" event={"ID":"0611b20f-b263-4167-a7c7-75e045e9fac4","Type":"ContainerStarted","Data":"23822e748d110f76a13a3880a4d4c0312f67486fa19dec68d2e98381120be2ea"} Aug 04 18:40:43.279374 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:43.279305 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" Aug 04 18:40:43.307854 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:43.307803 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" podStartSLOduration=1.624225094 podStartE2EDuration="3.307788101s" totalImagesPullingTime="1.683563007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:40:43.306380234 +0000 UTC m=+1800.557662794" watchObservedRunningTime="2026-08-04 18:40:43.307788101 +0000 UTC m=+1800.559070647" Aug 04 18:40:43.331750 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:43.331707 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" podStartSLOduration=2.331693038 podStartE2EDuration="2.331693038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:40:43.331545322 +0000 UTC m=+1800.582827870" watchObservedRunningTime="2026-08-04 18:40:43.331693038 +0000 UTC m=+1800.582975584" Aug 04 18:40:49.289235 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:49.289208 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" Aug 04 18:40:54.256276 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:54.256242 2526 patch_prober.go:28] interesting pod/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 18:40:54.256620 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:54.256329 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" podUID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 18:40:54.285411 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:54.285389 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" Aug 04 18:40:54.337150 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:54.337119 2526 generic.go:356] "Generic (PLEG): container finished" podID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerID="5ae26888c5386e123238e54ca09d83ce21efd026630bede8f9e143b1176d6074" exitCode=0 Aug 04 18:40:54.337288 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:54.337197 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" event={"ID":"803fee5d-ae6c-4309-a1bc-6d03be17c4a8","Type":"ContainerDied","Data":"5ae26888c5386e123238e54ca09d83ce21efd026630bede8f9e143b1176d6074"} Aug 04 18:40:54.337578 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:54.337561 2526 scope.go:117] "RemoveContainer" containerID="5ae26888c5386e123238e54ca09d83ce21efd026630bede8f9e143b1176d6074" Aug 04 18:40:54.338669 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:54.338634 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:40:54.338780 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:54.338762 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:40:54.338824 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:54.338812 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:40:55.343831 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:55.343795 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" event={"ID":"803fee5d-ae6c-4309-a1bc-6d03be17c4a8","Type":"ContainerStarted","Data":"d5d87a92c6d2eeb006cf20d08e6ad74caa9c6596cd6b885283660e7ba1243fbf"} Aug 04 18:40:56.168172 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:56.168133 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:40:56.168470 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:56.168442 2526 patch_prober.go:28] interesting pod/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 18:40:56.168572 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:56.168498 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" podUID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 18:40:57.168318 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:57.168278 2526 patch_prober.go:28] interesting pod/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 18:40:57.168909 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:57.168345 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" podUID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 18:40:58.168433 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:58.168393 2526 patch_prober.go:28] interesting pod/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 18:40:58.168912 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:58.168471 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" podUID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 18:40:59.167523 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:59.167490 2526 patch_prober.go:28] interesting pod/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 18:40:59.168027 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:40:59.167992 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" podUID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 18:41:00.168517 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:00.168473 2526 patch_prober.go:28] interesting pod/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 18:41:00.168915 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:00.168561 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" podUID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 18:41:01.171393 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:01.171362 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:41:01.171767 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:01.171622 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:41:01.172355 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:01.172338 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:41:11.694834 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:11.694797 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk"] Aug 04 18:41:11.695307 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:11.695067 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" podUID="ffbca186-f04a-4787-9127-8e86b8021f2f" containerName="maas-api" containerID="cri-o://2bf7d5191ed63de634de263ee901a36099e20d8c6d8458c17e5fce81bf24402d" gracePeriod=30 Aug 04 18:41:11.727224 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:11.727197 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:11.786895 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:11.786864 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz"] Aug 04 18:41:11.787112 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:11.787082 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" podUID="0611b20f-b263-4167-a7c7-75e045e9fac4" containerName="payload-processing" containerID="cri-o://74277d0d94c40eb09b143c3bca60cb0a8f28ae6b70c93a34339c4e6b20f2e8e2" gracePeriod=30 Aug 04 18:41:12.044374 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.044346 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" Aug 04 18:41:12.076139 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.076112 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" Aug 04 18:41:12.081215 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.081192 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffbca186-f04a-4787-9127-8e86b8021f2f-maas-api-tls\") pod \"ffbca186-f04a-4787-9127-8e86b8021f2f\" (UID: \"ffbca186-f04a-4787-9127-8e86b8021f2f\") " Aug 04 18:41:12.081356 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.081228 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbca186-f04a-4787-9127-8e86b8021f2f-openshift-service-ca\") pod \"ffbca186-f04a-4787-9127-8e86b8021f2f\" (UID: \"ffbca186-f04a-4787-9127-8e86b8021f2f\") " Aug 04 18:41:12.081356 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.081299 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwdgt\" (UniqueName: \"kubernetes.io/projected/ffbca186-f04a-4787-9127-8e86b8021f2f-kube-api-access-gwdgt\") pod \"ffbca186-f04a-4787-9127-8e86b8021f2f\" (UID: \"ffbca186-f04a-4787-9127-8e86b8021f2f\") " Aug 04 18:41:12.081670 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.081644 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffbca186-f04a-4787-9127-8e86b8021f2f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ffbca186-f04a-4787-9127-8e86b8021f2f" (UID: "ffbca186-f04a-4787-9127-8e86b8021f2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:41:12.083512 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.083482 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffbca186-f04a-4787-9127-8e86b8021f2f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ffbca186-f04a-4787-9127-8e86b8021f2f" (UID: "ffbca186-f04a-4787-9127-8e86b8021f2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:41:12.083601 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.083524 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffbca186-f04a-4787-9127-8e86b8021f2f-kube-api-access-gwdgt" (OuterVolumeSpecName: "kube-api-access-gwdgt") pod "ffbca186-f04a-4787-9127-8e86b8021f2f" (UID: "ffbca186-f04a-4787-9127-8e86b8021f2f"). InnerVolumeSpecName "kube-api-access-gwdgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:41:12.182753 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.182724 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcvcq\" (UniqueName: \"kubernetes.io/projected/0611b20f-b263-4167-a7c7-75e045e9fac4-kube-api-access-rcvcq\") pod \"0611b20f-b263-4167-a7c7-75e045e9fac4\" (UID: \"0611b20f-b263-4167-a7c7-75e045e9fac4\") " Aug 04 18:41:12.183034 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.182758 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0611b20f-b263-4167-a7c7-75e045e9fac4-plugins-config-volume\") pod \"0611b20f-b263-4167-a7c7-75e045e9fac4\" (UID: \"0611b20f-b263-4167-a7c7-75e045e9fac4\") " Aug 04 18:41:12.183034 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.182942 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwdgt\" (UniqueName: \"kubernetes.io/projected/ffbca186-f04a-4787-9127-8e86b8021f2f-kube-api-access-gwdgt\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:12.183034 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.182960 2526 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffbca186-f04a-4787-9127-8e86b8021f2f-maas-api-tls\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:12.183034 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.183002 2526 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbca186-f04a-4787-9127-8e86b8021f2f-openshift-service-ca\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:12.183204 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.183127 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0611b20f-b263-4167-a7c7-75e045e9fac4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0611b20f-b263-4167-a7c7-75e045e9fac4" (UID: "0611b20f-b263-4167-a7c7-75e045e9fac4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:41:12.184751 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.184728 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0611b20f-b263-4167-a7c7-75e045e9fac4-kube-api-access-rcvcq" (OuterVolumeSpecName: "kube-api-access-rcvcq") pod "0611b20f-b263-4167-a7c7-75e045e9fac4" (UID: "0611b20f-b263-4167-a7c7-75e045e9fac4"). InnerVolumeSpecName "kube-api-access-rcvcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:41:12.277411 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.277331 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb"] Aug 04 18:41:12.277591 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.277552 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" podUID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerName="istio-proxy" containerID="cri-o://d5d87a92c6d2eeb006cf20d08e6ad74caa9c6596cd6b885283660e7ba1243fbf" gracePeriod=30 Aug 04 18:41:12.284505 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.284477 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcvcq\" (UniqueName: \"kubernetes.io/projected/0611b20f-b263-4167-a7c7-75e045e9fac4-kube-api-access-rcvcq\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:12.284505 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.284504 2526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0611b20f-b263-4167-a7c7-75e045e9fac4-plugins-config-volume\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:12.442188 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.442152 2526 generic.go:356] "Generic (PLEG): container finished" podID="0611b20f-b263-4167-a7c7-75e045e9fac4" containerID="74277d0d94c40eb09b143c3bca60cb0a8f28ae6b70c93a34339c4e6b20f2e8e2" exitCode=0 Aug 04 18:41:12.442349 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.442210 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" Aug 04 18:41:12.442349 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.442269 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" event={"ID":"0611b20f-b263-4167-a7c7-75e045e9fac4","Type":"ContainerDied","Data":"74277d0d94c40eb09b143c3bca60cb0a8f28ae6b70c93a34339c4e6b20f2e8e2"} Aug 04 18:41:12.442349 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.442321 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz" event={"ID":"0611b20f-b263-4167-a7c7-75e045e9fac4","Type":"ContainerDied","Data":"23822e748d110f76a13a3880a4d4c0312f67486fa19dec68d2e98381120be2ea"} Aug 04 18:41:12.442349 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.442347 2526 scope.go:117] "RemoveContainer" containerID="74277d0d94c40eb09b143c3bca60cb0a8f28ae6b70c93a34339c4e6b20f2e8e2" Aug 04 18:41:12.443716 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.443693 2526 generic.go:356] "Generic (PLEG): container finished" podID="ffbca186-f04a-4787-9127-8e86b8021f2f" containerID="2bf7d5191ed63de634de263ee901a36099e20d8c6d8458c17e5fce81bf24402d" exitCode=0 Aug 04 18:41:12.443852 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.443721 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" event={"ID":"ffbca186-f04a-4787-9127-8e86b8021f2f","Type":"ContainerDied","Data":"2bf7d5191ed63de634de263ee901a36099e20d8c6d8458c17e5fce81bf24402d"} Aug 04 18:41:12.443852 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.443751 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" event={"ID":"ffbca186-f04a-4787-9127-8e86b8021f2f","Type":"ContainerDied","Data":"c326ed28ee6b04078196b79688022e5a5152341a659916b3e45b53df295b2474"} Aug 04 18:41:12.443852 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.443755 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk" Aug 04 18:41:12.456759 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.456742 2526 scope.go:117] "RemoveContainer" containerID="74277d0d94c40eb09b143c3bca60cb0a8f28ae6b70c93a34339c4e6b20f2e8e2" Aug 04 18:41:12.457017 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:41:12.457000 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74277d0d94c40eb09b143c3bca60cb0a8f28ae6b70c93a34339c4e6b20f2e8e2\": container with ID starting with 74277d0d94c40eb09b143c3bca60cb0a8f28ae6b70c93a34339c4e6b20f2e8e2 not found: ID does not exist" containerID="74277d0d94c40eb09b143c3bca60cb0a8f28ae6b70c93a34339c4e6b20f2e8e2" Aug 04 18:41:12.457065 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.457024 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74277d0d94c40eb09b143c3bca60cb0a8f28ae6b70c93a34339c4e6b20f2e8e2"} err="failed to get container status \"74277d0d94c40eb09b143c3bca60cb0a8f28ae6b70c93a34339c4e6b20f2e8e2\": rpc error: code = NotFound desc = could not find container \"74277d0d94c40eb09b143c3bca60cb0a8f28ae6b70c93a34339c4e6b20f2e8e2\": container with ID starting with 74277d0d94c40eb09b143c3bca60cb0a8f28ae6b70c93a34339c4e6b20f2e8e2 not found: ID does not exist" Aug 04 18:41:12.457065 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.457040 2526 scope.go:117] "RemoveContainer" containerID="2bf7d5191ed63de634de263ee901a36099e20d8c6d8458c17e5fce81bf24402d" Aug 04 18:41:12.464375 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.464360 2526 scope.go:117] "RemoveContainer" containerID="2bf7d5191ed63de634de263ee901a36099e20d8c6d8458c17e5fce81bf24402d" Aug 04 18:41:12.464610 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:41:12.464593 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf7d5191ed63de634de263ee901a36099e20d8c6d8458c17e5fce81bf24402d\": container with ID starting with 2bf7d5191ed63de634de263ee901a36099e20d8c6d8458c17e5fce81bf24402d not found: ID does not exist" containerID="2bf7d5191ed63de634de263ee901a36099e20d8c6d8458c17e5fce81bf24402d" Aug 04 18:41:12.464681 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.464618 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf7d5191ed63de634de263ee901a36099e20d8c6d8458c17e5fce81bf24402d"} err="failed to get container status \"2bf7d5191ed63de634de263ee901a36099e20d8c6d8458c17e5fce81bf24402d\": rpc error: code = NotFound desc = could not find container \"2bf7d5191ed63de634de263ee901a36099e20d8c6d8458c17e5fce81bf24402d\": container with ID starting with 2bf7d5191ed63de634de263ee901a36099e20d8c6d8458c17e5fce81bf24402d not found: ID does not exist" Aug 04 18:41:12.490898 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.490876 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz"] Aug 04 18:41:12.496210 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.496191 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9756c1e3-7d486bf98b-jtncz"] Aug 04 18:41:12.521619 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.521600 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk"] Aug 04 18:41:12.525623 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:12.525601 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9756c1e3-fdccdb7bf-zkxwk"] Aug 04 18:41:13.181100 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.181019 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0611b20f-b263-4167-a7c7-75e045e9fac4" path="/var/lib/kubelet/pods/0611b20f-b263-4167-a7c7-75e045e9fac4/volumes" Aug 04 18:41:13.181440 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.181396 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffbca186-f04a-4787-9127-8e86b8021f2f" path="/var/lib/kubelet/pods/ffbca186-f04a-4787-9127-8e86b8021f2f/volumes" Aug 04 18:41:13.483776 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.483745 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:13.607647 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.607611 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm"] Aug 04 18:41:13.608053 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.608031 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0611b20f-b263-4167-a7c7-75e045e9fac4" containerName="payload-processing" Aug 04 18:41:13.608053 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.608053 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="0611b20f-b263-4167-a7c7-75e045e9fac4" containerName="payload-processing" Aug 04 18:41:13.608254 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.608085 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffbca186-f04a-4787-9127-8e86b8021f2f" containerName="maas-api" Aug 04 18:41:13.608254 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.608092 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffbca186-f04a-4787-9127-8e86b8021f2f" containerName="maas-api" Aug 04 18:41:13.608254 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.608152 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffbca186-f04a-4787-9127-8e86b8021f2f" containerName="maas-api" Aug 04 18:41:13.608254 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.608167 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="0611b20f-b263-4167-a7c7-75e045e9fac4" containerName="payload-processing" Aug 04 18:41:13.612629 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.612611 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.649739 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.649714 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm"] Aug 04 18:41:13.658184 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:41:13.658003 2526 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-migrate-cf4512d1-gw-openshift-default-dockercfg-jlfr9\" is forbidden: User \"system:node:ip-10-0-136-165.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-165.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cf4512d1-gw-openshift-default-dockercfg-jlfr9\"" type="*v1.Secret" Aug 04 18:41:13.658184 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:41:13.658117 2526 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" is forbidden: User \"system:node:ip-10-0-136-165.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-165.ec2.internal' and this object" podUID="da39b81c-31af-437d-ab6d-8ae89812ee63" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.698855 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.698822 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-credential-socket\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.699087 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.698875 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da39b81c-31af-437d-ab6d-8ae89812ee63-istiod-ca-cert\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.699087 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.698934 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-token\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.699087 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.699025 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trwbr\" (UniqueName: \"kubernetes.io/projected/da39b81c-31af-437d-ab6d-8ae89812ee63-kube-api-access-trwbr\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.699087 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.699066 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-workload-socket\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.699322 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.699110 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-data\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.699322 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.699179 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-envoy\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.699322 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.699268 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-workload-certs\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.699322 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.699301 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-podinfo\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.730462 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.730430 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:13.800392 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.800308 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-token\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.800392 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.800352 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trwbr\" (UniqueName: \"kubernetes.io/projected/da39b81c-31af-437d-ab6d-8ae89812ee63-kube-api-access-trwbr\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.800392 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.800373 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-workload-socket\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.800625 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.800405 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-data\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.800625 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.800430 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-envoy\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.800625 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.800490 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-workload-certs\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.800625 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.800514 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-podinfo\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.800625 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.800555 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-credential-socket\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.800625 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.800612 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da39b81c-31af-437d-ab6d-8ae89812ee63-istiod-ca-cert\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.800925 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.800907 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-workload-socket\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.801013 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.800944 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-data\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.801068 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.801009 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-workload-certs\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.801109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.801075 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-credential-socket\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.801288 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.801272 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da39b81c-31af-437d-ab6d-8ae89812ee63-istiod-ca-cert\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.803343 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.803322 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-envoy\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.803457 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.803442 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-podinfo\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.841476 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.841455 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-token\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:13.841601 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:13.841584 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trwbr\" (UniqueName: \"kubernetes.io/projected/da39b81c-31af-437d-ab6d-8ae89812ee63-kube-api-access-trwbr\") pod \"e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:14.041633 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:14.041589 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:14.923403 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:14.923368 2526 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" secret="" err="failed to sync secret cache: timed out waiting for the condition" Aug 04 18:41:14.923818 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:14.923451 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:15.072684 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:15.072660 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm"] Aug 04 18:41:15.075178 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:41:15.075149 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda39b81c_31af_437d_ab6d_8ae89812ee63.slice/crio-ed83a303ebaa36b6685bf67b938a3ccce4a027a5414e8339f759128fdd9a809e WatchSource:0}: Error finding container ed83a303ebaa36b6685bf67b938a3ccce4a027a5414e8339f759128fdd9a809e: Status 404 returned error can't find the container with id ed83a303ebaa36b6685bf67b938a3ccce4a027a5414e8339f759128fdd9a809e Aug 04 18:41:15.077404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:15.077373 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:41:15.077537 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:15.077440 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:41:15.077537 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:15.077468 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:41:15.190485 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:15.190421 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cf4512d1-gw-openshift-default-dockercfg-jlfr9\"" Aug 04 18:41:15.462653 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:15.462621 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" event={"ID":"da39b81c-31af-437d-ab6d-8ae89812ee63","Type":"ContainerStarted","Data":"c171e82de0622fda0cbe344f3330399fb4508daa28396ecf2243b83490ab9507"} Aug 04 18:41:15.462791 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:15.462671 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" event={"ID":"da39b81c-31af-437d-ab6d-8ae89812ee63","Type":"ContainerStarted","Data":"ed83a303ebaa36b6685bf67b938a3ccce4a027a5414e8339f759128fdd9a809e"} Aug 04 18:41:15.487784 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:15.487729 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" podStartSLOduration=2.487711757 podStartE2EDuration="2.487711757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:41:15.487683359 +0000 UTC m=+1832.738965907" watchObservedRunningTime="2026-08-04 18:41:15.487711757 +0000 UTC m=+1832.738994305" Aug 04 18:41:15.923818 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:15.923740 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:16.173875 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:16.173796 2526 patch_prober.go:28] interesting pod/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 18:41:16.173875 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:16.173855 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" podUID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:41:16.929260 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:16.929232 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:17.105169 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.105117 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:17.413030 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.412997 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:17.477015 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.476600 2526 generic.go:356] "Generic (PLEG): container finished" podID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerID="d5d87a92c6d2eeb006cf20d08e6ad74caa9c6596cd6b885283660e7ba1243fbf" exitCode=0 Aug 04 18:41:17.477864 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.477826 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" event={"ID":"803fee5d-ae6c-4309-a1bc-6d03be17c4a8","Type":"ContainerDied","Data":"d5d87a92c6d2eeb006cf20d08e6ad74caa9c6596cd6b885283660e7ba1243fbf"} Aug 04 18:41:17.479555 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.479426 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:17.479555 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.479470 2526 scope.go:117] "RemoveContainer" containerID="5ae26888c5386e123238e54ca09d83ce21efd026630bede8f9e143b1176d6074" Aug 04 18:41:17.481307 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.481274 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:17.545486 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.545432 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:41:17.587622 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.587595 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:17.633319 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.633291 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-workload-certs\") pod \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " Aug 04 18:41:17.633438 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.633330 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkpht\" (UniqueName: \"kubernetes.io/projected/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-kube-api-access-nkpht\") pod \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " Aug 04 18:41:17.633438 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.633348 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-token\") pod \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " Aug 04 18:41:17.633438 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.633391 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-workload-socket\") pod \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " Aug 04 18:41:17.633594 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.633441 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-podinfo\") pod \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " Aug 04 18:41:17.633594 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.633467 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-credential-socket\") pod \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " Aug 04 18:41:17.633594 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.633494 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istiod-ca-cert\") pod \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " Aug 04 18:41:17.633594 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.633535 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-envoy\") pod \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " Aug 04 18:41:17.633594 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.633559 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-data\") pod \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\" (UID: \"803fee5d-ae6c-4309-a1bc-6d03be17c4a8\") " Aug 04 18:41:17.633890 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.633599 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "803fee5d-ae6c-4309-a1bc-6d03be17c4a8" (UID: "803fee5d-ae6c-4309-a1bc-6d03be17c4a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:41:17.633890 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.633730 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "803fee5d-ae6c-4309-a1bc-6d03be17c4a8" (UID: "803fee5d-ae6c-4309-a1bc-6d03be17c4a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:41:17.633890 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.633780 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "803fee5d-ae6c-4309-a1bc-6d03be17c4a8" (UID: "803fee5d-ae6c-4309-a1bc-6d03be17c4a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:41:17.634119 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.634031 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "803fee5d-ae6c-4309-a1bc-6d03be17c4a8" (UID: "803fee5d-ae6c-4309-a1bc-6d03be17c4a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:41:17.634119 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.634102 2526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-workload-certs\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:17.634216 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.634123 2526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-workload-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:17.634216 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.634139 2526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-credential-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:17.634216 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.634101 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-data" (OuterVolumeSpecName: "istio-data") pod "803fee5d-ae6c-4309-a1bc-6d03be17c4a8" (UID: "803fee5d-ae6c-4309-a1bc-6d03be17c4a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:41:17.635719 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.635696 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "803fee5d-ae6c-4309-a1bc-6d03be17c4a8" (UID: "803fee5d-ae6c-4309-a1bc-6d03be17c4a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:41:17.635719 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.635704 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-token" (OuterVolumeSpecName: "istio-token") pod "803fee5d-ae6c-4309-a1bc-6d03be17c4a8" (UID: "803fee5d-ae6c-4309-a1bc-6d03be17c4a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:41:17.635913 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.635898 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-kube-api-access-nkpht" (OuterVolumeSpecName: "kube-api-access-nkpht") pod "803fee5d-ae6c-4309-a1bc-6d03be17c4a8" (UID: "803fee5d-ae6c-4309-a1bc-6d03be17c4a8"). InnerVolumeSpecName "kube-api-access-nkpht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:41:17.636009 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.635987 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "803fee5d-ae6c-4309-a1bc-6d03be17c4a8" (UID: "803fee5d-ae6c-4309-a1bc-6d03be17c4a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:41:17.734718 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.734690 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkpht\" (UniqueName: \"kubernetes.io/projected/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-kube-api-access-nkpht\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:17.734718 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.734713 2526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-token\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:17.734718 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.734722 2526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-podinfo\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:17.734915 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.734732 2526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istiod-ca-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:17.734915 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.734741 2526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-envoy\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:17.734915 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:17.734748 2526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803fee5d-ae6c-4309-a1bc-6d03be17c4a8-istio-data\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:18.241582 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:18.241542 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:18.343100 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:18.343069 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:18.483513 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:18.483471 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" event={"ID":"803fee5d-ae6c-4309-a1bc-6d03be17c4a8","Type":"ContainerDied","Data":"0044e8171d7e2ae40c9090c186ae7fa32f93bccd735a51d7d6e091a8e85b7ecd"} Aug 04 18:41:18.483692 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:18.483550 2526 scope.go:117] "RemoveContainer" containerID="d5d87a92c6d2eeb006cf20d08e6ad74caa9c6596cd6b885283660e7ba1243fbf" Aug 04 18:41:18.483692 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:18.483606 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb" Aug 04 18:41:18.523362 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:18.523333 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb"] Aug 04 18:41:18.532432 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:18.532405 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9756c1e3-openshift-default-6758b9cc4b-vmmbb"] Aug 04 18:41:19.181731 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:19.181698 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" path="/var/lib/kubelet/pods/803fee5d-ae6c-4309-a1bc-6d03be17c4a8/volumes" Aug 04 18:41:30.544407 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:30.544374 2526 generic.go:356] "Generic (PLEG): container finished" podID="da39b81c-31af-437d-ab6d-8ae89812ee63" containerID="c171e82de0622fda0cbe344f3330399fb4508daa28396ecf2243b83490ab9507" exitCode=0 Aug 04 18:41:30.544803 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:30.544450 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" event={"ID":"da39b81c-31af-437d-ab6d-8ae89812ee63","Type":"ContainerDied","Data":"c171e82de0622fda0cbe344f3330399fb4508daa28396ecf2243b83490ab9507"} Aug 04 18:41:30.544803 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:30.544793 2526 scope.go:117] "RemoveContainer" containerID="c171e82de0622fda0cbe344f3330399fb4508daa28396ecf2243b83490ab9507" Aug 04 18:41:30.545778 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:30.545751 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:41:30.545836 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:30.545822 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:41:30.545871 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:30.545860 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:41:31.555222 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:31.555177 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" event={"ID":"da39b81c-31af-437d-ab6d-8ae89812ee63","Type":"ContainerStarted","Data":"1bb83f1de8e3f8f9e12222d6adf939a1b68df5f0cb48d833b0676be29703fd83"} Aug 04 18:41:31.924084 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:31.924002 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:31.925015 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:31.924991 2526 patch_prober.go:28] interesting pod/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Aug 04 18:41:31.925111 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:31.925046 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" podUID="da39b81c-31af-437d-ab6d-8ae89812ee63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Aug 04 18:41:32.479057 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:32.479023 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:32.924516 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:32.924429 2526 patch_prober.go:28] interesting pod/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Aug 04 18:41:32.924516 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:32.924484 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" podUID="da39b81c-31af-437d-ab6d-8ae89812ee63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Aug 04 18:41:33.924705 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:33.924668 2526 patch_prober.go:28] interesting pod/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Aug 04 18:41:33.925213 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:33.924734 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" podUID="da39b81c-31af-437d-ab6d-8ae89812ee63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Aug 04 18:41:34.924749 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:34.924713 2526 patch_prober.go:28] interesting pod/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Aug 04 18:41:34.925161 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:34.924770 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" podUID="da39b81c-31af-437d-ab6d-8ae89812ee63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Aug 04 18:41:35.924628 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:35.924593 2526 patch_prober.go:28] interesting pod/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Aug 04 18:41:35.924778 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:35.924648 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" podUID="da39b81c-31af-437d-ab6d-8ae89812ee63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Aug 04 18:41:36.928328 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:36.928296 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:36.929348 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:36.929327 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:49.225610 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:49.225572 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm"] Aug 04 18:41:49.226076 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:49.225826 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" podUID="da39b81c-31af-437d-ab6d-8ae89812ee63" containerName="istio-proxy" containerID="cri-o://1bb83f1de8e3f8f9e12222d6adf939a1b68df5f0cb48d833b0676be29703fd83" gracePeriod=30 Aug 04 18:41:49.457843 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:49.457810 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:49.552593 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:49.552492 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:49.781114 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:49.781076 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:50.072349 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.072318 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47"] Aug 04 18:41:50.072836 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.072818 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerName="istio-proxy" Aug 04 18:41:50.072836 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.072838 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerName="istio-proxy" Aug 04 18:41:50.072951 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.072849 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerName="istio-proxy" Aug 04 18:41:50.072951 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.072855 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerName="istio-proxy" Aug 04 18:41:50.072951 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.072911 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerName="istio-proxy" Aug 04 18:41:50.072951 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.072920 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="803fee5d-ae6c-4309-a1bc-6d03be17c4a8" containerName="istio-proxy" Aug 04 18:41:50.076000 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.075975 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.079142 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.079123 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3f6c154b-openshift-default-dockercfg-47s9g\"" Aug 04 18:41:50.098634 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.098605 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47"] Aug 04 18:41:50.122231 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.122206 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-envoy\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.122387 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.122239 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:50.122387 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.122257 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-podinfo\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.122387 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.122306 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-credential-socket\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.122387 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.122352 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-workload-socket\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.122534 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.122398 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-token\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.122534 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.122418 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-workload-certs\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.122534 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.122437 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c3b97d-a845-4460-bb94-6a02af04874a-istiod-ca-cert\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.122534 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.122464 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-data\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.122534 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.122497 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc64l\" (UniqueName: \"kubernetes.io/projected/f0c3b97d-a845-4460-bb94-6a02af04874a-kube-api-access-qc64l\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.223648 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.223610 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-podinfo\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.223827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.223665 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-credential-socket\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.223827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.223706 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-workload-socket\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.223827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.223733 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-token\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.223827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.223751 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-workload-certs\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.223827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.223772 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c3b97d-a845-4460-bb94-6a02af04874a-istiod-ca-cert\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.223827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.223809 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-data\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.224183 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.223852 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc64l\" (UniqueName: \"kubernetes.io/projected/f0c3b97d-a845-4460-bb94-6a02af04874a-kube-api-access-qc64l\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.224183 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.223885 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-envoy\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.224305 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.224190 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-credential-socket\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.224305 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.224201 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-workload-socket\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.224305 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.224276 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-data\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.224510 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.224486 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-workload-certs\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.225176 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.225157 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c3b97d-a845-4460-bb94-6a02af04874a-istiod-ca-cert\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.226112 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.226092 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-envoy\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.226428 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.226224 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-podinfo\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.235278 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.235250 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc64l\" (UniqueName: \"kubernetes.io/projected/f0c3b97d-a845-4460-bb94-6a02af04874a-kube-api-access-qc64l\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.235278 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.235258 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-token\") pod \"e2e-ait-3f6c154b-openshift-default-577d7f888-99r47\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.271225 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.271179 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:50.386502 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.386421 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:50.532420 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.532392 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47"] Aug 04 18:41:50.533641 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:41:50.533608 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c3b97d_a845_4460_bb94_6a02af04874a.slice/crio-28887f23a6fb8e36886534b3eb9682f743fbfd692d6bf3fa71b6a87c144bbd92 WatchSource:0}: Error finding container 28887f23a6fb8e36886534b3eb9682f743fbfd692d6bf3fa71b6a87c144bbd92: Status 404 returned error can't find the container with id 28887f23a6fb8e36886534b3eb9682f743fbfd692d6bf3fa71b6a87c144bbd92 Aug 04 18:41:50.535712 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.535682 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:41:50.535786 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.535746 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:41:50.535786 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.535772 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:41:50.657576 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.657492 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" event={"ID":"f0c3b97d-a845-4460-bb94-6a02af04874a","Type":"ContainerStarted","Data":"36b281d6aba5c961474d09dc2d1aa4cba6e7da925b19b3eaf4292db21fe6ef99"} Aug 04 18:41:50.657715 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.657583 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" event={"ID":"f0c3b97d-a845-4460-bb94-6a02af04874a","Type":"ContainerStarted","Data":"28887f23a6fb8e36886534b3eb9682f743fbfd692d6bf3fa71b6a87c144bbd92"} Aug 04 18:41:50.693635 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:50.693576 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" podStartSLOduration=0.693556038 podStartE2EDuration="693.556038ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:41:50.689911619 +0000 UTC m=+1867.941194188" watchObservedRunningTime="2026-08-04 18:41:50.693556038 +0000 UTC m=+1867.944838587" Aug 04 18:41:51.386616 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:51.386584 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:51.392047 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:51.392021 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:51.662117 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:51.662042 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:51.662997 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:51.662956 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:41:51.930770 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:51.930684 2526 patch_prober.go:28] interesting pod/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 18:41:51.930770 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:51.930748 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" podUID="da39b81c-31af-437d-ab6d-8ae89812ee63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:41:52.133097 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:52.133057 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:52.289952 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:52.289920 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:53.352311 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:53.352276 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:53.702742 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:53.702712 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:54.004475 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.004375 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:54.321615 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.321536 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:41:54.481869 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.481841 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:54.562430 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.562401 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da39b81c-31af-437d-ab6d-8ae89812ee63-istiod-ca-cert\") pod \"da39b81c-31af-437d-ab6d-8ae89812ee63\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " Aug 04 18:41:54.562625 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.562461 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-podinfo\") pod \"da39b81c-31af-437d-ab6d-8ae89812ee63\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " Aug 04 18:41:54.562625 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.562493 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-token\") pod \"da39b81c-31af-437d-ab6d-8ae89812ee63\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " Aug 04 18:41:54.562749 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.562655 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-data\") pod \"da39b81c-31af-437d-ab6d-8ae89812ee63\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " Aug 04 18:41:54.562749 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.562708 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-credential-socket\") pod \"da39b81c-31af-437d-ab6d-8ae89812ee63\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " Aug 04 18:41:54.562749 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.562744 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-workload-socket\") pod \"da39b81c-31af-437d-ab6d-8ae89812ee63\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " Aug 04 18:41:54.562901 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.562772 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-envoy\") pod \"da39b81c-31af-437d-ab6d-8ae89812ee63\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " Aug 04 18:41:54.562901 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.562802 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trwbr\" (UniqueName: \"kubernetes.io/projected/da39b81c-31af-437d-ab6d-8ae89812ee63-kube-api-access-trwbr\") pod \"da39b81c-31af-437d-ab6d-8ae89812ee63\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " Aug 04 18:41:54.562901 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.562816 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da39b81c-31af-437d-ab6d-8ae89812ee63-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da39b81c-31af-437d-ab6d-8ae89812ee63" (UID: "da39b81c-31af-437d-ab6d-8ae89812ee63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:41:54.562901 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.562859 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-workload-certs\") pod \"da39b81c-31af-437d-ab6d-8ae89812ee63\" (UID: \"da39b81c-31af-437d-ab6d-8ae89812ee63\") " Aug 04 18:41:54.563132 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.563106 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da39b81c-31af-437d-ab6d-8ae89812ee63" (UID: "da39b81c-31af-437d-ab6d-8ae89812ee63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:41:54.563253 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.563234 2526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da39b81c-31af-437d-ab6d-8ae89812ee63-istiod-ca-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:54.563324 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.563285 2526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-workload-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:54.563379 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.563333 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da39b81c-31af-437d-ab6d-8ae89812ee63" (UID: "da39b81c-31af-437d-ab6d-8ae89812ee63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:41:54.563379 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.563366 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-data" (OuterVolumeSpecName: "istio-data") pod "da39b81c-31af-437d-ab6d-8ae89812ee63" (UID: "da39b81c-31af-437d-ab6d-8ae89812ee63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:41:54.563630 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.563567 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da39b81c-31af-437d-ab6d-8ae89812ee63" (UID: "da39b81c-31af-437d-ab6d-8ae89812ee63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:41:54.565225 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.565190 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da39b81c-31af-437d-ab6d-8ae89812ee63" (UID: "da39b81c-31af-437d-ab6d-8ae89812ee63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:41:54.565315 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.565232 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da39b81c-31af-437d-ab6d-8ae89812ee63" (UID: "da39b81c-31af-437d-ab6d-8ae89812ee63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:41:54.565315 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.565246 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-token" (OuterVolumeSpecName: "istio-token") pod "da39b81c-31af-437d-ab6d-8ae89812ee63" (UID: "da39b81c-31af-437d-ab6d-8ae89812ee63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:41:54.565767 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.565742 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da39b81c-31af-437d-ab6d-8ae89812ee63-kube-api-access-trwbr" (OuterVolumeSpecName: "kube-api-access-trwbr") pod "da39b81c-31af-437d-ab6d-8ae89812ee63" (UID: "da39b81c-31af-437d-ab6d-8ae89812ee63"). InnerVolumeSpecName "kube-api-access-trwbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:41:54.664168 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.664139 2526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-workload-certs\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:54.664168 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.664163 2526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-podinfo\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:54.664168 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.664171 2526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-token\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:54.664393 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.664180 2526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-data\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:54.664393 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.664188 2526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-credential-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:54.664393 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.664196 2526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da39b81c-31af-437d-ab6d-8ae89812ee63-istio-envoy\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:54.664393 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.664206 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trwbr\" (UniqueName: \"kubernetes.io/projected/da39b81c-31af-437d-ab6d-8ae89812ee63-kube-api-access-trwbr\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:41:54.678659 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.678628 2526 generic.go:356] "Generic (PLEG): container finished" podID="da39b81c-31af-437d-ab6d-8ae89812ee63" containerID="1bb83f1de8e3f8f9e12222d6adf939a1b68df5f0cb48d833b0676be29703fd83" exitCode=0 Aug 04 18:41:54.678807 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.678675 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" event={"ID":"da39b81c-31af-437d-ab6d-8ae89812ee63","Type":"ContainerDied","Data":"1bb83f1de8e3f8f9e12222d6adf939a1b68df5f0cb48d833b0676be29703fd83"} Aug 04 18:41:54.678807 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.678686 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" Aug 04 18:41:54.678807 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.678731 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm" event={"ID":"da39b81c-31af-437d-ab6d-8ae89812ee63","Type":"ContainerDied","Data":"ed83a303ebaa36b6685bf67b938a3ccce4a027a5414e8339f759128fdd9a809e"} Aug 04 18:41:54.678807 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.678758 2526 scope.go:117] "RemoveContainer" containerID="1bb83f1de8e3f8f9e12222d6adf939a1b68df5f0cb48d833b0676be29703fd83" Aug 04 18:41:54.687916 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.687898 2526 scope.go:117] "RemoveContainer" containerID="c171e82de0622fda0cbe344f3330399fb4508daa28396ecf2243b83490ab9507" Aug 04 18:41:54.695757 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.695736 2526 scope.go:117] "RemoveContainer" containerID="1bb83f1de8e3f8f9e12222d6adf939a1b68df5f0cb48d833b0676be29703fd83" Aug 04 18:41:54.696031 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:41:54.696009 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb83f1de8e3f8f9e12222d6adf939a1b68df5f0cb48d833b0676be29703fd83\": container with ID starting with 1bb83f1de8e3f8f9e12222d6adf939a1b68df5f0cb48d833b0676be29703fd83 not found: ID does not exist" containerID="1bb83f1de8e3f8f9e12222d6adf939a1b68df5f0cb48d833b0676be29703fd83" Aug 04 18:41:54.696105 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.696039 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb83f1de8e3f8f9e12222d6adf939a1b68df5f0cb48d833b0676be29703fd83"} err="failed to get container status \"1bb83f1de8e3f8f9e12222d6adf939a1b68df5f0cb48d833b0676be29703fd83\": rpc error: code = NotFound desc = could not find container \"1bb83f1de8e3f8f9e12222d6adf939a1b68df5f0cb48d833b0676be29703fd83\": container with ID starting with 1bb83f1de8e3f8f9e12222d6adf939a1b68df5f0cb48d833b0676be29703fd83 not found: ID does not exist" Aug 04 18:41:54.696105 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.696056 2526 scope.go:117] "RemoveContainer" containerID="c171e82de0622fda0cbe344f3330399fb4508daa28396ecf2243b83490ab9507" Aug 04 18:41:54.696336 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:41:54.696315 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c171e82de0622fda0cbe344f3330399fb4508daa28396ecf2243b83490ab9507\": container with ID starting with c171e82de0622fda0cbe344f3330399fb4508daa28396ecf2243b83490ab9507 not found: ID does not exist" containerID="c171e82de0622fda0cbe344f3330399fb4508daa28396ecf2243b83490ab9507" Aug 04 18:41:54.696402 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.696346 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c171e82de0622fda0cbe344f3330399fb4508daa28396ecf2243b83490ab9507"} err="failed to get container status \"c171e82de0622fda0cbe344f3330399fb4508daa28396ecf2243b83490ab9507\": rpc error: code = NotFound desc = could not find container \"c171e82de0622fda0cbe344f3330399fb4508daa28396ecf2243b83490ab9507\": container with ID starting with c171e82de0622fda0cbe344f3330399fb4508daa28396ecf2243b83490ab9507 not found: ID does not exist" Aug 04 18:41:54.703460 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.703440 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm"] Aug 04 18:41:54.711083 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:54.711061 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cf4512d1-gw-openshift-default-575564cb68-xnjfm"] Aug 04 18:41:55.182224 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:41:55.182196 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da39b81c-31af-437d-ab6d-8ae89812ee63" path="/var/lib/kubelet/pods/da39b81c-31af-437d-ab6d-8ae89812ee63/volumes" Aug 04 18:42:06.663387 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:06.663353 2526 patch_prober.go:28] interesting pod/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:42:06.663804 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:06.663419 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" podUID="f0c3b97d-a845-4460-bb94-6a02af04874a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 18:42:06.744906 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:06.744875 2526 generic.go:356] "Generic (PLEG): container finished" podID="f0c3b97d-a845-4460-bb94-6a02af04874a" containerID="36b281d6aba5c961474d09dc2d1aa4cba6e7da925b19b3eaf4292db21fe6ef99" exitCode=0 Aug 04 18:42:06.745092 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:06.744942 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" event={"ID":"f0c3b97d-a845-4460-bb94-6a02af04874a","Type":"ContainerDied","Data":"36b281d6aba5c961474d09dc2d1aa4cba6e7da925b19b3eaf4292db21fe6ef99"} Aug 04 18:42:06.745386 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:06.745371 2526 scope.go:117] "RemoveContainer" containerID="36b281d6aba5c961474d09dc2d1aa4cba6e7da925b19b3eaf4292db21fe6ef99" Aug 04 18:42:06.746411 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:06.746379 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:42:06.746504 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:06.746487 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:42:06.746548 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:06.746525 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:42:07.755167 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:07.755126 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" event={"ID":"f0c3b97d-a845-4460-bb94-6a02af04874a","Type":"ContainerStarted","Data":"11b3efddb8e557cd83736be8fbf276b361d1eade00c2499c477c4a45e23f41b9"} Aug 04 18:42:08.387038 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:08.387004 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:42:08.387265 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:08.387245 2526 patch_prober.go:28] interesting pod/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:42:08.387354 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:08.387310 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" podUID="f0c3b97d-a845-4460-bb94-6a02af04874a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 18:42:09.387383 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:09.387340 2526 patch_prober.go:28] interesting pod/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:42:09.387860 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:09.387403 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" podUID="f0c3b97d-a845-4460-bb94-6a02af04874a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 18:42:10.387618 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:10.387577 2526 patch_prober.go:28] interesting pod/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:42:10.388072 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:10.387637 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" podUID="f0c3b97d-a845-4460-bb94-6a02af04874a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 18:42:11.386833 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:11.386795 2526 patch_prober.go:28] interesting pod/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:42:11.387055 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:11.386862 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" podUID="f0c3b97d-a845-4460-bb94-6a02af04874a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 18:42:12.387134 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:12.387098 2526 patch_prober.go:28] interesting pod/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:42:12.387497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:12.387167 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" podUID="f0c3b97d-a845-4460-bb94-6a02af04874a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 18:42:13.390651 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:13.390620 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:42:13.391040 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:13.390995 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:42:13.391655 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:13.391637 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:42:34.127209 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:34.127174 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:34.542480 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:34.542439 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:34.652127 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:34.652090 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:35.711877 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:35.711836 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47"] Aug 04 18:42:35.712326 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:35.712137 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" podUID="f0c3b97d-a845-4460-bb94-6a02af04874a" containerName="istio-proxy" containerID="cri-o://11b3efddb8e557cd83736be8fbf276b361d1eade00c2499c477c4a45e23f41b9" gracePeriod=30 Aug 04 18:42:35.744310 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:35.744282 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:35.975702 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:35.975619 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:36.220359 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.220326 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:36.533117 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.533086 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv"] Aug 04 18:42:36.533500 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.533487 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da39b81c-31af-437d-ab6d-8ae89812ee63" containerName="istio-proxy" Aug 04 18:42:36.533612 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.533502 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="da39b81c-31af-437d-ab6d-8ae89812ee63" containerName="istio-proxy" Aug 04 18:42:36.533612 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.533517 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da39b81c-31af-437d-ab6d-8ae89812ee63" containerName="istio-proxy" Aug 04 18:42:36.533612 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.533524 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="da39b81c-31af-437d-ab6d-8ae89812ee63" containerName="istio-proxy" Aug 04 18:42:36.533612 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.533608 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="da39b81c-31af-437d-ab6d-8ae89812ee63" containerName="istio-proxy" Aug 04 18:42:36.533813 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.533617 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="da39b81c-31af-437d-ab6d-8ae89812ee63" containerName="istio-proxy" Aug 04 18:42:36.536662 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.536642 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.539336 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.539320 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-16e32844-openshift-default-dockercfg-5p8w4\"" Aug 04 18:42:36.551163 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.551138 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv"] Aug 04 18:42:36.583031 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.583003 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:36.647949 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.647911 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2r2l\" (UniqueName: \"kubernetes.io/projected/c292bdcd-1438-4241-9c89-d7c745965bc9-kube-api-access-z2r2l\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.648156 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.647997 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-credential-socket\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.648156 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.648028 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-data\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.648156 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.648065 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-workload-socket\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.648156 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.648110 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-workload-certs\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.648375 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.648158 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-podinfo\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.648375 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.648219 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-token\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.648375 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.648313 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-envoy\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.648375 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.648357 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c292bdcd-1438-4241-9c89-d7c745965bc9-istiod-ca-cert\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.749801 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.749762 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2r2l\" (UniqueName: \"kubernetes.io/projected/c292bdcd-1438-4241-9c89-d7c745965bc9-kube-api-access-z2r2l\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.749801 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.749805 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-credential-socket\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.750345 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.749828 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-data\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.750345 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.749915 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-workload-socket\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.750345 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.749957 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-workload-certs\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.750345 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.749994 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-podinfo\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.750345 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.750050 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-token\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.750345 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.750116 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-envoy\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.750345 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.750158 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c292bdcd-1438-4241-9c89-d7c745965bc9-istiod-ca-cert\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.750345 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.750214 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-data\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.750345 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.750325 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-credential-socket\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.750755 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.750691 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-workload-socket\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.750755 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.750712 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-workload-certs\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.750839 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.750793 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c292bdcd-1438-4241-9c89-d7c745965bc9-istiod-ca-cert\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.752558 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.752540 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-envoy\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.752729 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.752707 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-podinfo\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.759645 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.759597 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2r2l\" (UniqueName: \"kubernetes.io/projected/c292bdcd-1438-4241-9c89-d7c745965bc9-kube-api-access-z2r2l\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.759907 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.759886 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-token\") pod \"e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.847470 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.847392 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:36.992452 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.992424 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv"] Aug 04 18:42:36.993902 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:42:36.993875 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc292bdcd_1438_4241_9c89_d7c745965bc9.slice/crio-7e0e97fe698ee390d8b7631856f07398549df06b0b94f1ef731767486ced491c WatchSource:0}: Error finding container 7e0e97fe698ee390d8b7631856f07398549df06b0b94f1ef731767486ced491c: Status 404 returned error can't find the container with id 7e0e97fe698ee390d8b7631856f07398549df06b0b94f1ef731767486ced491c Aug 04 18:42:36.996086 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.996055 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:42:36.996169 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.996120 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:42:36.996169 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:36.996145 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:42:37.079621 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:37.079594 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:37.905943 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:37.905911 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" event={"ID":"c292bdcd-1438-4241-9c89-d7c745965bc9","Type":"ContainerStarted","Data":"f86b3ca1dfbad64fc00e62dba8ac7a41f801d301b8be53045197a233ddc48522"} Aug 04 18:42:37.906329 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:37.905960 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" event={"ID":"c292bdcd-1438-4241-9c89-d7c745965bc9","Type":"ContainerStarted","Data":"7e0e97fe698ee390d8b7631856f07398549df06b0b94f1ef731767486ced491c"} Aug 04 18:42:37.931514 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:37.931465 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" podStartSLOduration=1.931451772 podStartE2EDuration="1.931451772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:42:37.929893632 +0000 UTC m=+1915.181176179" watchObservedRunningTime="2026-08-04 18:42:37.931451772 +0000 UTC m=+1915.182734318" Aug 04 18:42:38.848226 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:38.848190 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:38.852987 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:38.852939 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:38.911299 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:38.911274 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:38.912101 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:38.912085 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:39.290564 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:39.290524 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:39.390994 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:39.390945 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:40.505579 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:40.505547 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:40.861441 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:40.861395 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:40.927239 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:40.927210 2526 generic.go:356] "Generic (PLEG): container finished" podID="f0c3b97d-a845-4460-bb94-6a02af04874a" containerID="11b3efddb8e557cd83736be8fbf276b361d1eade00c2499c477c4a45e23f41b9" exitCode=0 Aug 04 18:42:40.927444 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:40.927282 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" event={"ID":"f0c3b97d-a845-4460-bb94-6a02af04874a","Type":"ContainerDied","Data":"11b3efddb8e557cd83736be8fbf276b361d1eade00c2499c477c4a45e23f41b9"} Aug 04 18:42:40.927444 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:40.927349 2526 scope.go:117] "RemoveContainer" containerID="36b281d6aba5c961474d09dc2d1aa4cba6e7da925b19b3eaf4292db21fe6ef99" Aug 04 18:42:40.969627 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:40.969603 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:42:41.080895 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.080808 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:41.088960 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.088939 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-podinfo\") pod \"f0c3b97d-a845-4460-bb94-6a02af04874a\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " Aug 04 18:42:41.089048 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.088999 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-credential-socket\") pod \"f0c3b97d-a845-4460-bb94-6a02af04874a\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " Aug 04 18:42:41.089048 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.089018 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-workload-socket\") pod \"f0c3b97d-a845-4460-bb94-6a02af04874a\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " Aug 04 18:42:41.089122 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.089080 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-envoy\") pod \"f0c3b97d-a845-4460-bb94-6a02af04874a\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " Aug 04 18:42:41.089122 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.089096 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-workload-certs\") pod \"f0c3b97d-a845-4460-bb94-6a02af04874a\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " Aug 04 18:42:41.089218 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.089128 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-token\") pod \"f0c3b97d-a845-4460-bb94-6a02af04874a\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " Aug 04 18:42:41.089218 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.089178 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c3b97d-a845-4460-bb94-6a02af04874a-istiod-ca-cert\") pod \"f0c3b97d-a845-4460-bb94-6a02af04874a\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " Aug 04 18:42:41.089218 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.089205 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qc64l\" (UniqueName: \"kubernetes.io/projected/f0c3b97d-a845-4460-bb94-6a02af04874a-kube-api-access-qc64l\") pod \"f0c3b97d-a845-4460-bb94-6a02af04874a\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " Aug 04 18:42:41.089341 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.089230 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-data\") pod \"f0c3b97d-a845-4460-bb94-6a02af04874a\" (UID: \"f0c3b97d-a845-4460-bb94-6a02af04874a\") " Aug 04 18:42:41.089341 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.089254 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0c3b97d-a845-4460-bb94-6a02af04874a" (UID: "f0c3b97d-a845-4460-bb94-6a02af04874a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:42:41.089341 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.089287 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0c3b97d-a845-4460-bb94-6a02af04874a" (UID: "f0c3b97d-a845-4460-bb94-6a02af04874a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:42:41.090005 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.089586 2526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-credential-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:42:41.090005 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.089613 2526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-workload-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:42:41.090005 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.089726 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0c3b97d-a845-4460-bb94-6a02af04874a" (UID: "f0c3b97d-a845-4460-bb94-6a02af04874a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:42:41.090005 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.089929 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0c3b97d-a845-4460-bb94-6a02af04874a" (UID: "f0c3b97d-a845-4460-bb94-6a02af04874a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:42:41.090310 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.090031 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0c3b97d-a845-4460-bb94-6a02af04874a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0c3b97d-a845-4460-bb94-6a02af04874a" (UID: "f0c3b97d-a845-4460-bb94-6a02af04874a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:42:41.091707 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.091662 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c3b97d-a845-4460-bb94-6a02af04874a-kube-api-access-qc64l" (OuterVolumeSpecName: "kube-api-access-qc64l") pod "f0c3b97d-a845-4460-bb94-6a02af04874a" (UID: "f0c3b97d-a845-4460-bb94-6a02af04874a"). InnerVolumeSpecName "kube-api-access-qc64l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:42:41.091804 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.091707 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0c3b97d-a845-4460-bb94-6a02af04874a" (UID: "f0c3b97d-a845-4460-bb94-6a02af04874a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:42:41.091804 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.091724 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0c3b97d-a845-4460-bb94-6a02af04874a" (UID: "f0c3b97d-a845-4460-bb94-6a02af04874a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:42:41.091890 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.091818 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0c3b97d-a845-4460-bb94-6a02af04874a" (UID: "f0c3b97d-a845-4460-bb94-6a02af04874a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:42:41.183635 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.183585 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:41.190912 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.190890 2526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-envoy\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:42:41.190912 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.190913 2526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-workload-certs\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:42:41.191094 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.190925 2526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-token\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:42:41.191094 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.190937 2526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c3b97d-a845-4460-bb94-6a02af04874a-istiod-ca-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:42:41.191094 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.190948 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc64l\" (UniqueName: \"kubernetes.io/projected/f0c3b97d-a845-4460-bb94-6a02af04874a-kube-api-access-qc64l\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:42:41.191094 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.190979 2526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-data\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:42:41.191094 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.190994 2526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c3b97d-a845-4460-bb94-6a02af04874a-istio-podinfo\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:42:41.933364 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.933326 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" event={"ID":"f0c3b97d-a845-4460-bb94-6a02af04874a","Type":"ContainerDied","Data":"28887f23a6fb8e36886534b3eb9682f743fbfd692d6bf3fa71b6a87c144bbd92"} Aug 04 18:42:41.933364 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.933362 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47" Aug 04 18:42:41.933845 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.933399 2526 scope.go:117] "RemoveContainer" containerID="11b3efddb8e557cd83736be8fbf276b361d1eade00c2499c477c4a45e23f41b9" Aug 04 18:42:41.952536 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.952501 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47"] Aug 04 18:42:41.957397 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:41.957367 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3f6c154b-openshift-default-577d7f888-99r47"] Aug 04 18:42:43.181333 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:43.181300 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0c3b97d-a845-4460-bb94-6a02af04874a" path="/var/lib/kubelet/pods/f0c3b97d-a845-4460-bb94-6a02af04874a/volumes" Aug 04 18:42:53.912729 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:53.912691 2526 patch_prober.go:28] interesting pod/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 18:42:53.913172 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:53.912759 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" podUID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 18:42:54.006758 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:54.006727 2526 generic.go:356] "Generic (PLEG): container finished" podID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerID="f86b3ca1dfbad64fc00e62dba8ac7a41f801d301b8be53045197a233ddc48522" exitCode=0 Aug 04 18:42:54.006927 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:54.006804 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" event={"ID":"c292bdcd-1438-4241-9c89-d7c745965bc9","Type":"ContainerDied","Data":"f86b3ca1dfbad64fc00e62dba8ac7a41f801d301b8be53045197a233ddc48522"} Aug 04 18:42:54.007260 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:54.007246 2526 scope.go:117] "RemoveContainer" containerID="f86b3ca1dfbad64fc00e62dba8ac7a41f801d301b8be53045197a233ddc48522" Aug 04 18:42:54.008284 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:54.008248 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:42:54.008355 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:54.008324 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:42:54.008398 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:54.008362 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:42:55.013391 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:55.013345 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" event={"ID":"c292bdcd-1438-4241-9c89-d7c745965bc9","Type":"ContainerStarted","Data":"7823f830afeec35c7aac16638eb7713c64ce6ce18ff073e7a2131439fc347836"} Aug 04 18:42:55.848444 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:55.848413 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:42:55.848789 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:55.848766 2526 patch_prober.go:28] interesting pod/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 18:42:55.848957 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:55.848929 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" podUID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 18:42:56.848587 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:56.848538 2526 patch_prober.go:28] interesting pod/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 18:42:56.849214 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:56.849179 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" podUID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 18:42:57.848168 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:57.848132 2526 patch_prober.go:28] interesting pod/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 18:42:57.848505 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:57.848459 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" podUID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 18:42:58.848634 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:58.848598 2526 patch_prober.go:28] interesting pod/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 18:42:58.849249 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:58.849218 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" podUID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 18:42:59.847919 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:59.847872 2526 patch_prober.go:28] interesting pod/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 18:42:59.848149 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:42:59.847943 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" podUID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 18:43:00.851880 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:00.851845 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:43:00.852298 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:00.852203 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:43:00.852814 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:00.852795 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:43:11.254398 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:11.254363 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:11.345811 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:11.345777 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:11.419534 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:11.419502 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:11.491929 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:11.491871 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:11.629745 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:11.629656 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv"] Aug 04 18:43:11.630009 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:11.629895 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" podUID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerName="istio-proxy" containerID="cri-o://7823f830afeec35c7aac16638eb7713c64ce6ce18ff073e7a2131439fc347836" gracePeriod=30 Aug 04 18:43:11.654957 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:11.654931 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:11.913148 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:11.913066 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:12.413923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:12.413887 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:12.494446 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:12.494413 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:12.583827 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:12.583794 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:15.854876 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:15.854830 2526 patch_prober.go:28] interesting pod/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Aug 04 18:43:15.855296 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:15.854936 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" podUID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:43:16.887751 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:16.887725 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:43:17.040282 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.040245 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2r2l\" (UniqueName: \"kubernetes.io/projected/c292bdcd-1438-4241-9c89-d7c745965bc9-kube-api-access-z2r2l\") pod \"c292bdcd-1438-4241-9c89-d7c745965bc9\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " Aug 04 18:43:17.040282 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.040289 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-token\") pod \"c292bdcd-1438-4241-9c89-d7c745965bc9\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " Aug 04 18:43:17.040552 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.040321 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c292bdcd-1438-4241-9c89-d7c745965bc9-istiod-ca-cert\") pod \"c292bdcd-1438-4241-9c89-d7c745965bc9\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " Aug 04 18:43:17.040552 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.040460 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-workload-socket\") pod \"c292bdcd-1438-4241-9c89-d7c745965bc9\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " Aug 04 18:43:17.040552 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.040499 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-workload-certs\") pod \"c292bdcd-1438-4241-9c89-d7c745965bc9\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " Aug 04 18:43:17.040552 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.040536 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-envoy\") pod \"c292bdcd-1438-4241-9c89-d7c745965bc9\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " Aug 04 18:43:17.040766 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.040597 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-credential-socket\") pod \"c292bdcd-1438-4241-9c89-d7c745965bc9\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " Aug 04 18:43:17.040766 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.040675 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-podinfo\") pod \"c292bdcd-1438-4241-9c89-d7c745965bc9\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " Aug 04 18:43:17.040766 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.040756 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-data\") pod \"c292bdcd-1438-4241-9c89-d7c745965bc9\" (UID: \"c292bdcd-1438-4241-9c89-d7c745965bc9\") " Aug 04 18:43:17.040920 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.040760 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c292bdcd-1438-4241-9c89-d7c745965bc9" (UID: "c292bdcd-1438-4241-9c89-d7c745965bc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:43:17.040920 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.040779 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c292bdcd-1438-4241-9c89-d7c745965bc9" (UID: "c292bdcd-1438-4241-9c89-d7c745965bc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:43:17.040920 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.040819 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c292bdcd-1438-4241-9c89-d7c745965bc9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c292bdcd-1438-4241-9c89-d7c745965bc9" (UID: "c292bdcd-1438-4241-9c89-d7c745965bc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:43:17.041161 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.041056 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c292bdcd-1438-4241-9c89-d7c745965bc9" (UID: "c292bdcd-1438-4241-9c89-d7c745965bc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:43:17.041161 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.041075 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-data" (OuterVolumeSpecName: "istio-data") pod "c292bdcd-1438-4241-9c89-d7c745965bc9" (UID: "c292bdcd-1438-4241-9c89-d7c745965bc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:43:17.041273 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.041162 2526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c292bdcd-1438-4241-9c89-d7c745965bc9-istiod-ca-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:43:17.041273 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.041175 2526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-workload-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:43:17.041273 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.041185 2526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-workload-certs\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:43:17.041273 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.041199 2526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-credential-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:43:17.041273 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.041210 2526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-data\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:43:17.043164 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.043135 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c292bdcd-1438-4241-9c89-d7c745965bc9-kube-api-access-z2r2l" (OuterVolumeSpecName: "kube-api-access-z2r2l") pod "c292bdcd-1438-4241-9c89-d7c745965bc9" (UID: "c292bdcd-1438-4241-9c89-d7c745965bc9"). InnerVolumeSpecName "kube-api-access-z2r2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:43:17.043311 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.043202 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-token" (OuterVolumeSpecName: "istio-token") pod "c292bdcd-1438-4241-9c89-d7c745965bc9" (UID: "c292bdcd-1438-4241-9c89-d7c745965bc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:43:17.043311 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.043305 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c292bdcd-1438-4241-9c89-d7c745965bc9" (UID: "c292bdcd-1438-4241-9c89-d7c745965bc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:43:17.043430 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.043407 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c292bdcd-1438-4241-9c89-d7c745965bc9" (UID: "c292bdcd-1438-4241-9c89-d7c745965bc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:43:17.119462 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.119430 2526 generic.go:356] "Generic (PLEG): container finished" podID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerID="7823f830afeec35c7aac16638eb7713c64ce6ce18ff073e7a2131439fc347836" exitCode=0 Aug 04 18:43:17.119626 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.119480 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" Aug 04 18:43:17.119626 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.119513 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" event={"ID":"c292bdcd-1438-4241-9c89-d7c745965bc9","Type":"ContainerDied","Data":"7823f830afeec35c7aac16638eb7713c64ce6ce18ff073e7a2131439fc347836"} Aug 04 18:43:17.119626 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.119575 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv" event={"ID":"c292bdcd-1438-4241-9c89-d7c745965bc9","Type":"ContainerDied","Data":"7e0e97fe698ee390d8b7631856f07398549df06b0b94f1ef731767486ced491c"} Aug 04 18:43:17.119626 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.119599 2526 scope.go:117] "RemoveContainer" containerID="7823f830afeec35c7aac16638eb7713c64ce6ce18ff073e7a2131439fc347836" Aug 04 18:43:17.128876 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.128860 2526 scope.go:117] "RemoveContainer" containerID="f86b3ca1dfbad64fc00e62dba8ac7a41f801d301b8be53045197a233ddc48522" Aug 04 18:43:17.136357 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.136340 2526 scope.go:117] "RemoveContainer" containerID="7823f830afeec35c7aac16638eb7713c64ce6ce18ff073e7a2131439fc347836" Aug 04 18:43:17.136594 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:43:17.136576 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7823f830afeec35c7aac16638eb7713c64ce6ce18ff073e7a2131439fc347836\": container with ID starting with 7823f830afeec35c7aac16638eb7713c64ce6ce18ff073e7a2131439fc347836 not found: ID does not exist" containerID="7823f830afeec35c7aac16638eb7713c64ce6ce18ff073e7a2131439fc347836" Aug 04 18:43:17.136694 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.136607 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7823f830afeec35c7aac16638eb7713c64ce6ce18ff073e7a2131439fc347836"} err="failed to get container status \"7823f830afeec35c7aac16638eb7713c64ce6ce18ff073e7a2131439fc347836\": rpc error: code = NotFound desc = could not find container \"7823f830afeec35c7aac16638eb7713c64ce6ce18ff073e7a2131439fc347836\": container with ID starting with 7823f830afeec35c7aac16638eb7713c64ce6ce18ff073e7a2131439fc347836 not found: ID does not exist" Aug 04 18:43:17.136694 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.136631 2526 scope.go:117] "RemoveContainer" containerID="f86b3ca1dfbad64fc00e62dba8ac7a41f801d301b8be53045197a233ddc48522" Aug 04 18:43:17.136866 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:43:17.136850 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86b3ca1dfbad64fc00e62dba8ac7a41f801d301b8be53045197a233ddc48522\": container with ID starting with f86b3ca1dfbad64fc00e62dba8ac7a41f801d301b8be53045197a233ddc48522 not found: ID does not exist" containerID="f86b3ca1dfbad64fc00e62dba8ac7a41f801d301b8be53045197a233ddc48522" Aug 04 18:43:17.136907 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.136871 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86b3ca1dfbad64fc00e62dba8ac7a41f801d301b8be53045197a233ddc48522"} err="failed to get container status \"f86b3ca1dfbad64fc00e62dba8ac7a41f801d301b8be53045197a233ddc48522\": rpc error: code = NotFound desc = could not find container \"f86b3ca1dfbad64fc00e62dba8ac7a41f801d301b8be53045197a233ddc48522\": container with ID starting with f86b3ca1dfbad64fc00e62dba8ac7a41f801d301b8be53045197a233ddc48522 not found: ID does not exist" Aug 04 18:43:17.142202 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.142186 2526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-podinfo\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:43:17.142289 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.142203 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2r2l\" (UniqueName: \"kubernetes.io/projected/c292bdcd-1438-4241-9c89-d7c745965bc9-kube-api-access-z2r2l\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:43:17.142289 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.142214 2526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-token\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:43:17.142289 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.142223 2526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c292bdcd-1438-4241-9c89-d7c745965bc9-istio-envoy\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:43:17.158541 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.158516 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv"] Aug 04 18:43:17.168982 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.168938 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-16e32844-openshift-default-9fcdc99f6-ghskv"] Aug 04 18:43:17.182529 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:17.182502 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c292bdcd-1438-4241-9c89-d7c745965bc9" path="/var/lib/kubelet/pods/c292bdcd-1438-4241-9c89-d7c745965bc9/volumes" Aug 04 18:43:24.376799 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:24.376761 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:32.044057 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:32.044015 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:32.151186 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:32.151153 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:32.267622 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:32.267593 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:32.354446 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:32.354365 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:32.805615 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:32.805587 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:58.843366 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:43:58.843334 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:05.777787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:05.777755 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:05.889861 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:05.889830 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:05.992784 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:05.992752 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:11.967205 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:11.967172 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:21.969486 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:21.969451 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:28.420472 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:28.420442 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:28.489642 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:28.489618 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:28.579475 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:28.579450 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:28.678489 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:28.678401 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:39.211417 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:39.211385 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:50.331730 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:50.331699 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:57.475113 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:57.475066 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:57.628257 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:57.628221 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:57.847098 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:57.847009 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:57.998300 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:57.998271 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:58.634330 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:58.634302 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:58.842676 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:44:58.842648 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:00.456510 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:00.456472 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:01.702260 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:01.702205 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:03.201206 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:03.201176 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:08.138481 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:08.138401 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:08.240194 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:08.240157 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:08.873396 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:08.873367 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:16.464994 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:16.464939 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:16.590600 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:16.590572 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:16.815621 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:16.815543 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:17.113114 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.113038 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt"] Aug 04 18:45:17.113408 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.113395 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0c3b97d-a845-4460-bb94-6a02af04874a" containerName="istio-proxy" Aug 04 18:45:17.113452 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.113409 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0c3b97d-a845-4460-bb94-6a02af04874a" containerName="istio-proxy" Aug 04 18:45:17.113452 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.113422 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerName="istio-proxy" Aug 04 18:45:17.113452 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.113427 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerName="istio-proxy" Aug 04 18:45:17.113452 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.113436 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0c3b97d-a845-4460-bb94-6a02af04874a" containerName="istio-proxy" Aug 04 18:45:17.113452 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.113442 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0c3b97d-a845-4460-bb94-6a02af04874a" containerName="istio-proxy" Aug 04 18:45:17.113452 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.113450 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerName="istio-proxy" Aug 04 18:45:17.113639 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.113456 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerName="istio-proxy" Aug 04 18:45:17.113639 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.113512 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerName="istio-proxy" Aug 04 18:45:17.113639 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.113520 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0c3b97d-a845-4460-bb94-6a02af04874a" containerName="istio-proxy" Aug 04 18:45:17.113639 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.113526 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="c292bdcd-1438-4241-9c89-d7c745965bc9" containerName="istio-proxy" Aug 04 18:45:17.113639 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.113533 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0c3b97d-a845-4460-bb94-6a02af04874a" containerName="istio-proxy" Aug 04 18:45:17.116744 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.116728 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.121783 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.121764 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f11a0dab-openshift-default-dockercfg-rv5tk\"" Aug 04 18:45:17.184538 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.184508 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt"] Aug 04 18:45:17.190145 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.190107 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rhm2\" (UniqueName: \"kubernetes.io/projected/98f2bbed-9c0d-4525-8329-15a5a59cefbb-kube-api-access-8rhm2\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.190145 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.190155 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-workload-socket\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.190359 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.190190 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-credential-socket\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.190359 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.190326 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-podinfo\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.190470 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.190375 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-token\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.190470 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.190427 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-data\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.190470 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.190456 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-workload-certs\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.190624 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.190491 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istiod-ca-cert\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.190624 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.190519 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-envoy\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.193266 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.193196 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:17.291777 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.291747 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-credential-socket\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.291929 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.291793 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-podinfo\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.291929 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.291817 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-token\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.291929 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.291864 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-data\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.291929 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.291901 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-workload-certs\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.292175 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.291936 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istiod-ca-cert\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.292175 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.291957 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-envoy\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.292175 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.292010 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rhm2\" (UniqueName: \"kubernetes.io/projected/98f2bbed-9c0d-4525-8329-15a5a59cefbb-kube-api-access-8rhm2\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.292175 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.292048 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-workload-socket\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.292383 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.292183 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-credential-socket\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.292383 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.292284 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-data\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.292383 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.292340 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-workload-certs\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.292546 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.292529 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-workload-socket\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.292665 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.292639 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istiod-ca-cert\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.294327 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.294307 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-envoy\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.294430 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.294413 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-podinfo\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.326581 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.326558 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rhm2\" (UniqueName: \"kubernetes.io/projected/98f2bbed-9c0d-4525-8329-15a5a59cefbb-kube-api-access-8rhm2\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.329766 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.329749 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-token\") pod \"e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.427207 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.427131 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:17.449604 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.449578 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:17.599176 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.599151 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt"] Aug 04 18:45:17.600002 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:45:17.599950 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f2bbed_9c0d_4525_8329_15a5a59cefbb.slice/crio-c041e8e3b285462ae71891390054a2b6edf0e91f6f6d846d60136c787fd8ef68 WatchSource:0}: Error finding container c041e8e3b285462ae71891390054a2b6edf0e91f6f6d846d60136c787fd8ef68: Status 404 returned error can't find the container with id c041e8e3b285462ae71891390054a2b6edf0e91f6f6d846d60136c787fd8ef68 Aug 04 18:45:17.602128 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.602084 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:45:17.602204 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.602172 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:45:17.602246 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.602203 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:45:17.714785 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.714749 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" event={"ID":"98f2bbed-9c0d-4525-8329-15a5a59cefbb","Type":"ContainerStarted","Data":"a935db893527d79c51d512f45afb8a158fbe9cde92d9cb3baba1c34585a4211a"} Aug 04 18:45:17.714992 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.714815 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" event={"ID":"98f2bbed-9c0d-4525-8329-15a5a59cefbb","Type":"ContainerStarted","Data":"c041e8e3b285462ae71891390054a2b6edf0e91f6f6d846d60136c787fd8ef68"} Aug 04 18:45:17.752927 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:17.752882 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" podStartSLOduration=0.752864916 podStartE2EDuration="752.864916ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:45:17.75149177 +0000 UTC m=+2075.002774317" watchObservedRunningTime="2026-08-04 18:45:17.752864916 +0000 UTC m=+2075.004147463" Aug 04 18:45:18.427844 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:18.427812 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:18.429433 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:18.429403 2526 patch_prober.go:28] interesting pod/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 18:45:18.429545 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:18.429453 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" podUID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 18:45:19.432046 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:19.432015 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:19.724442 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:19.724412 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:19.725405 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:19.725387 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:20.188640 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:20.188546 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:20.304903 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:20.304870 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:21.351796 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:21.351763 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:21.675362 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:21.675266 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:22.051941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:22.051911 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:22.179742 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:22.179714 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:34.726568 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:34.726492 2526 patch_prober.go:28] interesting pod/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 18:45:34.726568 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:34.726560 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" podUID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 18:45:34.804634 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:34.804605 2526 generic.go:356] "Generic (PLEG): container finished" podID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerID="a935db893527d79c51d512f45afb8a158fbe9cde92d9cb3baba1c34585a4211a" exitCode=0 Aug 04 18:45:34.804759 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:34.804677 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" event={"ID":"98f2bbed-9c0d-4525-8329-15a5a59cefbb","Type":"ContainerDied","Data":"a935db893527d79c51d512f45afb8a158fbe9cde92d9cb3baba1c34585a4211a"} Aug 04 18:45:34.805061 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:34.805042 2526 scope.go:117] "RemoveContainer" containerID="a935db893527d79c51d512f45afb8a158fbe9cde92d9cb3baba1c34585a4211a" Aug 04 18:45:34.806019 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:34.805991 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:45:34.806084 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:34.806059 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:45:34.806122 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:34.806091 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:45:35.811519 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:35.811487 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" event={"ID":"98f2bbed-9c0d-4525-8329-15a5a59cefbb","Type":"ContainerStarted","Data":"707f57e42da34f95869d8141f0d5d8f3152b108f27212d98d8b1cd700cb07e05"} Aug 04 18:45:36.427989 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:36.427933 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:36.428162 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:36.428145 2526 patch_prober.go:28] interesting pod/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 18:45:36.428210 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:36.428192 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" podUID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 18:45:37.427625 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:37.427587 2526 patch_prober.go:28] interesting pod/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 18:45:37.428044 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:37.427642 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" podUID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 18:45:38.427570 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:38.427539 2526 patch_prober.go:28] interesting pod/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 18:45:38.427743 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:38.427589 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" podUID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 18:45:39.428333 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:39.428297 2526 patch_prober.go:28] interesting pod/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 18:45:39.428692 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:39.428365 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" podUID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 18:45:40.428320 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:40.428286 2526 patch_prober.go:28] interesting pod/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 18:45:40.428504 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:40.428342 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" podUID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 18:45:41.431761 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:41.431732 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:41.432218 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:41.432077 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:41.432816 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:41.432798 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:45:50.340297 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:50.340264 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:51.483223 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:51.483194 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:51.582640 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:51.582612 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:52.372499 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:52.372466 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:53.201183 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:53.201150 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:53.311729 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:53.311693 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:58.844139 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:58.844103 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:58.945283 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:45:58.945252 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:09.826649 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:09.826619 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:09.926701 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:09.926662 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:10.030602 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:10.030569 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:10.132840 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:10.132752 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:10.511602 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:10.511562 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:18.020145 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:18.020106 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:18.366418 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:18.366341 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt"] Aug 04 18:46:18.366593 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:18.366556 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" podUID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerName="istio-proxy" containerID="cri-o://707f57e42da34f95869d8141f0d5d8f3152b108f27212d98d8b1cd700cb07e05" gracePeriod=30 Aug 04 18:46:21.736405 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:21.736372 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:23.612778 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.612754 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:46:23.666838 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.666760 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-credential-socket\") pod \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " Aug 04 18:46:23.666838 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.666836 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-workload-socket\") pod \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " Aug 04 18:46:23.667036 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.666882 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-token\") pod \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " Aug 04 18:46:23.667036 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.666913 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-data\") pod \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " Aug 04 18:46:23.667036 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.667018 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-envoy\") pod \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " Aug 04 18:46:23.667193 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.667067 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rhm2\" (UniqueName: \"kubernetes.io/projected/98f2bbed-9c0d-4525-8329-15a5a59cefbb-kube-api-access-8rhm2\") pod \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " Aug 04 18:46:23.667193 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.667093 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-workload-certs\") pod \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " Aug 04 18:46:23.667193 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.667115 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "98f2bbed-9c0d-4525-8329-15a5a59cefbb" (UID: "98f2bbed-9c0d-4525-8329-15a5a59cefbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:46:23.667193 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.667138 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-podinfo\") pod \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " Aug 04 18:46:23.667406 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.667196 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istiod-ca-cert\") pod \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\" (UID: \"98f2bbed-9c0d-4525-8329-15a5a59cefbb\") " Aug 04 18:46:23.667406 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.667212 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "98f2bbed-9c0d-4525-8329-15a5a59cefbb" (UID: "98f2bbed-9c0d-4525-8329-15a5a59cefbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:46:23.667406 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.667226 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-data" (OuterVolumeSpecName: "istio-data") pod "98f2bbed-9c0d-4525-8329-15a5a59cefbb" (UID: "98f2bbed-9c0d-4525-8329-15a5a59cefbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:46:23.667565 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.667450 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "98f2bbed-9c0d-4525-8329-15a5a59cefbb" (UID: "98f2bbed-9c0d-4525-8329-15a5a59cefbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:46:23.667565 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.667500 2526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-credential-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:46:23.667565 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.667518 2526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-workload-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:46:23.667565 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.667534 2526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-data\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:46:23.667782 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.667635 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "98f2bbed-9c0d-4525-8329-15a5a59cefbb" (UID: "98f2bbed-9c0d-4525-8329-15a5a59cefbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:46:23.669165 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.669142 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-token" (OuterVolumeSpecName: "istio-token") pod "98f2bbed-9c0d-4525-8329-15a5a59cefbb" (UID: "98f2bbed-9c0d-4525-8329-15a5a59cefbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:46:23.669376 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.669347 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "98f2bbed-9c0d-4525-8329-15a5a59cefbb" (UID: "98f2bbed-9c0d-4525-8329-15a5a59cefbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:46:23.669630 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.669611 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f2bbed-9c0d-4525-8329-15a5a59cefbb-kube-api-access-8rhm2" (OuterVolumeSpecName: "kube-api-access-8rhm2") pod "98f2bbed-9c0d-4525-8329-15a5a59cefbb" (UID: "98f2bbed-9c0d-4525-8329-15a5a59cefbb"). InnerVolumeSpecName "kube-api-access-8rhm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:46:23.669630 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.669621 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "98f2bbed-9c0d-4525-8329-15a5a59cefbb" (UID: "98f2bbed-9c0d-4525-8329-15a5a59cefbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:46:23.768013 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.767990 2526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-token\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:46:23.768013 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.768009 2526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-envoy\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:46:23.768013 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.768018 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rhm2\" (UniqueName: \"kubernetes.io/projected/98f2bbed-9c0d-4525-8329-15a5a59cefbb-kube-api-access-8rhm2\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:46:23.768262 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.768027 2526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f2bbed-9c0d-4525-8329-15a5a59cefbb-workload-certs\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:46:23.768262 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.768035 2526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istio-podinfo\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:46:23.768262 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:23.768052 2526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f2bbed-9c0d-4525-8329-15a5a59cefbb-istiod-ca-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:46:24.044399 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:24.044371 2526 generic.go:356] "Generic (PLEG): container finished" podID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerID="707f57e42da34f95869d8141f0d5d8f3152b108f27212d98d8b1cd700cb07e05" exitCode=0 Aug 04 18:46:24.044565 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:24.044442 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" Aug 04 18:46:24.044565 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:24.044454 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" event={"ID":"98f2bbed-9c0d-4525-8329-15a5a59cefbb","Type":"ContainerDied","Data":"707f57e42da34f95869d8141f0d5d8f3152b108f27212d98d8b1cd700cb07e05"} Aug 04 18:46:24.044565 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:24.044527 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt" event={"ID":"98f2bbed-9c0d-4525-8329-15a5a59cefbb","Type":"ContainerDied","Data":"c041e8e3b285462ae71891390054a2b6edf0e91f6f6d846d60136c787fd8ef68"} Aug 04 18:46:24.044565 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:24.044550 2526 scope.go:117] "RemoveContainer" containerID="707f57e42da34f95869d8141f0d5d8f3152b108f27212d98d8b1cd700cb07e05" Aug 04 18:46:24.055093 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:24.055074 2526 scope.go:117] "RemoveContainer" containerID="a935db893527d79c51d512f45afb8a158fbe9cde92d9cb3baba1c34585a4211a" Aug 04 18:46:24.064097 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:24.064082 2526 scope.go:117] "RemoveContainer" containerID="707f57e42da34f95869d8141f0d5d8f3152b108f27212d98d8b1cd700cb07e05" Aug 04 18:46:24.064357 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:46:24.064339 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707f57e42da34f95869d8141f0d5d8f3152b108f27212d98d8b1cd700cb07e05\": container with ID starting with 707f57e42da34f95869d8141f0d5d8f3152b108f27212d98d8b1cd700cb07e05 not found: ID does not exist" containerID="707f57e42da34f95869d8141f0d5d8f3152b108f27212d98d8b1cd700cb07e05" Aug 04 18:46:24.064430 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:24.064369 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707f57e42da34f95869d8141f0d5d8f3152b108f27212d98d8b1cd700cb07e05"} err="failed to get container status \"707f57e42da34f95869d8141f0d5d8f3152b108f27212d98d8b1cd700cb07e05\": rpc error: code = NotFound desc = could not find container \"707f57e42da34f95869d8141f0d5d8f3152b108f27212d98d8b1cd700cb07e05\": container with ID starting with 707f57e42da34f95869d8141f0d5d8f3152b108f27212d98d8b1cd700cb07e05 not found: ID does not exist" Aug 04 18:46:24.064430 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:24.064393 2526 scope.go:117] "RemoveContainer" containerID="a935db893527d79c51d512f45afb8a158fbe9cde92d9cb3baba1c34585a4211a" Aug 04 18:46:24.064639 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:46:24.064620 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a935db893527d79c51d512f45afb8a158fbe9cde92d9cb3baba1c34585a4211a\": container with ID starting with a935db893527d79c51d512f45afb8a158fbe9cde92d9cb3baba1c34585a4211a not found: ID does not exist" containerID="a935db893527d79c51d512f45afb8a158fbe9cde92d9cb3baba1c34585a4211a" Aug 04 18:46:24.064678 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:24.064644 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a935db893527d79c51d512f45afb8a158fbe9cde92d9cb3baba1c34585a4211a"} err="failed to get container status \"a935db893527d79c51d512f45afb8a158fbe9cde92d9cb3baba1c34585a4211a\": rpc error: code = NotFound desc = could not find container \"a935db893527d79c51d512f45afb8a158fbe9cde92d9cb3baba1c34585a4211a\": container with ID starting with a935db893527d79c51d512f45afb8a158fbe9cde92d9cb3baba1c34585a4211a not found: ID does not exist" Aug 04 18:46:24.073511 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:24.073491 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt"] Aug 04 18:46:24.085300 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:24.085281 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f11a0dab-openshift-default-7cc7bc5c44-22nwt"] Aug 04 18:46:25.181013 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:25.180983 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" path="/var/lib/kubelet/pods/98f2bbed-9c0d-4525-8329-15a5a59cefbb/volumes" Aug 04 18:46:27.416164 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:27.416121 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:27.499291 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:27.499257 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:27.602609 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:27.602574 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:27.699070 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:27.699042 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:27.807491 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:27.807464 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:35.017469 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:35.017375 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:35.145076 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:35.145032 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:52.178580 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:52.178546 2526 generic.go:356] "Generic (PLEG): container finished" podID="c6cc8554-0e4a-4b65-af23-661da2fcf0f3" containerID="be1668b8b7291d0a5efb1bd6960bd58a9e4b4e1cbc6eea4428808c11d96938a0" exitCode=1 Aug 04 18:46:52.179030 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:52.178621 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" event={"ID":"c6cc8554-0e4a-4b65-af23-661da2fcf0f3","Type":"ContainerDied","Data":"be1668b8b7291d0a5efb1bd6960bd58a9e4b4e1cbc6eea4428808c11d96938a0"} Aug 04 18:46:52.179119 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:52.179066 2526 scope.go:117] "RemoveContainer" containerID="be1668b8b7291d0a5efb1bd6960bd58a9e4b4e1cbc6eea4428808c11d96938a0" Aug 04 18:46:52.179428 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:52.179412 2526 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:46:52.180345 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:52.180325 2526 generic.go:356] "Generic (PLEG): container finished" podID="2256ff0e-84f0-4c89-a5e8-6809a4be5154" containerID="693faa6b7322064e441550c868f6286feb057466e9aa7d24d2cdd2eb109d9d10" exitCode=1 Aug 04 18:46:52.180422 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:52.180401 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk" event={"ID":"2256ff0e-84f0-4c89-a5e8-6809a4be5154","Type":"ContainerDied","Data":"693faa6b7322064e441550c868f6286feb057466e9aa7d24d2cdd2eb109d9d10"} Aug 04 18:46:52.180700 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:52.180687 2526 scope.go:117] "RemoveContainer" containerID="693faa6b7322064e441550c868f6286feb057466e9aa7d24d2cdd2eb109d9d10" Aug 04 18:46:53.186057 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:53.186020 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" event={"ID":"c6cc8554-0e4a-4b65-af23-661da2fcf0f3","Type":"ContainerStarted","Data":"804a3c33df1a0caa62a7a6aa06bfb44dbeed1424182345b5d94b9fd9b42c4ae0"} Aug 04 18:46:53.186491 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:53.186247 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" Aug 04 18:46:53.187593 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:53.187570 2526 generic.go:356] "Generic (PLEG): container finished" podID="e92e0caa-ebcb-4775-92c1-f66c63d895e0" containerID="e22e7a617dd23bc4582cf880e528e14d7c452b7cd5eb9ab82274804074433832" exitCode=1 Aug 04 18:46:53.187715 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:53.187639 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg" event={"ID":"e92e0caa-ebcb-4775-92c1-f66c63d895e0","Type":"ContainerDied","Data":"e22e7a617dd23bc4582cf880e528e14d7c452b7cd5eb9ab82274804074433832"} Aug 04 18:46:53.187868 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:53.187855 2526 scope.go:117] "RemoveContainer" containerID="e22e7a617dd23bc4582cf880e528e14d7c452b7cd5eb9ab82274804074433832" Aug 04 18:46:53.189716 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:53.189694 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk" event={"ID":"2256ff0e-84f0-4c89-a5e8-6809a4be5154","Type":"ContainerStarted","Data":"2d2b2e5b721b921223f25981b29a5298504c2e24e46c6a8c334d5e8bbd0d7cce"} Aug 04 18:46:53.189904 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:53.189886 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk" Aug 04 18:46:53.193656 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:53.193639 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Aug 04 18:46:54.196343 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:54.196310 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg" event={"ID":"e92e0caa-ebcb-4775-92c1-f66c63d895e0","Type":"ContainerStarted","Data":"6d393b4111883b35c35d5c453b078e794a0114db7a9450537ab87207d1ff7612"} Aug 04 18:46:54.196748 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:46:54.196723 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg" Aug 04 18:47:04.198914 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:04.198885 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mtwwk" Aug 04 18:47:04.199380 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:04.198947 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-fdvj6" Aug 04 18:47:05.204735 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:05.204702 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hl5wg" Aug 04 18:47:10.271939 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:10.271875 2526 generic.go:356] "Generic (PLEG): container finished" podID="1caa4745-97d7-40d8-b66b-ee647e71a995" containerID="28e9c8f6e45826f63499e4668a8160e80847c652c585413829378cd1a9ded3c7" exitCode=1 Aug 04 18:47:10.272286 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:10.271950 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" event={"ID":"1caa4745-97d7-40d8-b66b-ee647e71a995","Type":"ContainerDied","Data":"28e9c8f6e45826f63499e4668a8160e80847c652c585413829378cd1a9ded3c7"} Aug 04 18:47:10.272335 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:10.272288 2526 scope.go:117] "RemoveContainer" containerID="28e9c8f6e45826f63499e4668a8160e80847c652c585413829378cd1a9ded3c7" Aug 04 18:47:10.274076 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:10.274050 2526 generic.go:356] "Generic (PLEG): container finished" podID="eb341c5b-c651-4e86-a138-f9d2b0d747cf" containerID="1f2accf045c1dfbd49dcb7ccf70f4d02d7ea84f67b1e6a00ecbf92cf993e63a2" exitCode=1 Aug 04 18:47:10.274168 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:10.274131 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" event={"ID":"eb341c5b-c651-4e86-a138-f9d2b0d747cf","Type":"ContainerDied","Data":"1f2accf045c1dfbd49dcb7ccf70f4d02d7ea84f67b1e6a00ecbf92cf993e63a2"} Aug 04 18:47:10.274517 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:10.274437 2526 scope.go:117] "RemoveContainer" containerID="1f2accf045c1dfbd49dcb7ccf70f4d02d7ea84f67b1e6a00ecbf92cf993e63a2" Aug 04 18:47:10.275667 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:10.275644 2526 generic.go:356] "Generic (PLEG): container finished" podID="69af5af2-c017-4455-b51d-03fe560a578b" containerID="406d9dac6646b2439a5fd02b4e9f9a45d8d6641acaab7e19418230fba034d49c" exitCode=1 Aug 04 18:47:10.275757 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:10.275712 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" event={"ID":"69af5af2-c017-4455-b51d-03fe560a578b","Type":"ContainerDied","Data":"406d9dac6646b2439a5fd02b4e9f9a45d8d6641acaab7e19418230fba034d49c"} Aug 04 18:47:10.276002 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:10.275988 2526 scope.go:117] "RemoveContainer" containerID="406d9dac6646b2439a5fd02b4e9f9a45d8d6641acaab7e19418230fba034d49c" Aug 04 18:47:11.281947 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:11.281915 2526 generic.go:356] "Generic (PLEG): container finished" podID="a7e54ecf-6cca-42e8-a708-edc70403685f" containerID="395a2a9e98d0670cafa1a8cda982e3aa43a69964d6f777491d1133cd015cabd2" exitCode=1 Aug 04 18:47:11.282384 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:11.281987 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cffffc5dc-blwt8" event={"ID":"a7e54ecf-6cca-42e8-a708-edc70403685f","Type":"ContainerDied","Data":"395a2a9e98d0670cafa1a8cda982e3aa43a69964d6f777491d1133cd015cabd2"} Aug 04 18:47:11.282384 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:11.282314 2526 scope.go:117] "RemoveContainer" containerID="395a2a9e98d0670cafa1a8cda982e3aa43a69964d6f777491d1133cd015cabd2" Aug 04 18:47:11.283877 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:11.283857 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" event={"ID":"69af5af2-c017-4455-b51d-03fe560a578b","Type":"ContainerStarted","Data":"8848828a67b7d63dd8ebea04d7634e0387d40f05604c742a2e5f34906eefb7ac"} Aug 04 18:47:11.284119 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:11.284104 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" Aug 04 18:47:11.285501 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:11.285482 2526 generic.go:356] "Generic (PLEG): container finished" podID="23b21430-239b-4799-8119-ed05095d4ee9" containerID="9d3309a444a28ae086a7a025a53cb7153c28be82bc26498dbe79c62309be5c77" exitCode=1 Aug 04 18:47:11.285607 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:11.285550 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" event={"ID":"23b21430-239b-4799-8119-ed05095d4ee9","Type":"ContainerDied","Data":"9d3309a444a28ae086a7a025a53cb7153c28be82bc26498dbe79c62309be5c77"} Aug 04 18:47:11.285814 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:11.285798 2526 scope.go:117] "RemoveContainer" containerID="9d3309a444a28ae086a7a025a53cb7153c28be82bc26498dbe79c62309be5c77" Aug 04 18:47:11.287214 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:11.287198 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" event={"ID":"1caa4745-97d7-40d8-b66b-ee647e71a995","Type":"ContainerStarted","Data":"5bb9a6ae8c7283167cd766d5708fb7725bebe55abc1ccac816da6e51f975ed48"} Aug 04 18:47:11.287386 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:11.287368 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" Aug 04 18:47:11.290128 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:11.290108 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" event={"ID":"eb341c5b-c651-4e86-a138-f9d2b0d747cf","Type":"ContainerStarted","Data":"da74a6d278ec59e17b668f3c0518f4bddf70c05b176ab177df040e962cff8c8f"} Aug 04 18:47:11.290294 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:11.290275 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:47:12.115720 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:47:12.115630 2526 controller.go:195] "Failed to update lease" err="Put \"https://aeb9e022d670540fdaf7590f6abafc97-f9f04794ff9f3669.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-136-165.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Aug 04 18:47:12.302842 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:12.302805 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" event={"ID":"23b21430-239b-4799-8119-ed05095d4ee9","Type":"ContainerStarted","Data":"64274df66924eb233ce31bc4e122e24fc3f066909f18dfeeed7a678bdba34da6"} Aug 04 18:47:12.303392 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:12.303355 2526 status_manager.go:444] "Container readiness changed for unknown container" pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" containerID="cri-o://9d3309a444a28ae086a7a025a53cb7153c28be82bc26498dbe79c62309be5c77" Aug 04 18:47:12.303479 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:12.303398 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" Aug 04 18:47:12.863218 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:47:12.863177 2526 kuberuntime_manager.go:1459] "Unhandled Error" err="container manager start failed in pod maas-controller-5cffffc5dc-blwt8_opendatahub(a7e54ecf-6cca-42e8-a708-edc70403685f): CreateContainerConfigError: failed to sync configmap cache: timed out waiting for the condition" logger="UnhandledError" Aug 04 18:47:12.863388 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:47:12.863226 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CreateContainerConfigError: \"failed to sync configmap cache: timed out waiting for the condition\"" pod="opendatahub/maas-controller-5cffffc5dc-blwt8" podUID="a7e54ecf-6cca-42e8-a708-edc70403685f" Aug 04 18:47:13.308550 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:13.308523 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" Aug 04 18:47:14.936079 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:14.936046 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 18:47:19.226483 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:19.226450 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:19.860665 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:19.860632 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cffffc5dc-blwt8" Aug 04 18:47:19.860999 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:19.860960 2526 scope.go:117] "RemoveContainer" containerID="395a2a9e98d0670cafa1a8cda982e3aa43a69964d6f777491d1133cd015cabd2" Aug 04 18:47:19.861195 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:47:19.861176 2526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=maas-controller-5cffffc5dc-blwt8_opendatahub(a7e54ecf-6cca-42e8-a708-edc70403685f)\"" pod="opendatahub/maas-controller-5cffffc5dc-blwt8" podUID="a7e54ecf-6cca-42e8-a708-edc70403685f" Aug 04 18:47:19.910925 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:19.910900 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:21.027580 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:21.027551 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:22.304806 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:22.304778 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-s48rg" Aug 04 18:47:24.315788 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:24.315753 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-dmvk7" Aug 04 18:47:25.507230 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:25.507199 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:26.210351 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:26.210317 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:27.160381 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:27.160350 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:28.048420 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:28.048373 2526 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/maas-controller-5cffffc5dc-blwt8" Aug 04 18:47:28.048780 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:28.048763 2526 scope.go:117] "RemoveContainer" containerID="395a2a9e98d0670cafa1a8cda982e3aa43a69964d6f777491d1133cd015cabd2" Aug 04 18:47:29.391169 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:29.391134 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cffffc5dc-blwt8" event={"ID":"a7e54ecf-6cca-42e8-a708-edc70403685f","Type":"ContainerStarted","Data":"474d5c8bd2622804775ede83ba8326cb018f936a89f2e7801fa9fb9792ef916f"} Aug 04 18:47:29.391548 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:29.391346 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cffffc5dc-blwt8" Aug 04 18:47:35.289509 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:35.289474 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:36.415246 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:36.415216 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:40.398046 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:40.398014 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cffffc5dc-blwt8" Aug 04 18:47:42.304123 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:42.304091 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7j4tr" Aug 04 18:47:42.307580 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:42.307559 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2cdvs" Aug 04 18:47:44.654335 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:44.654294 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:45.461493 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:45.461467 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:46.408842 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:46.408805 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:47.490278 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:47.490239 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:52.317592 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:52.317562 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:52.460280 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:52.460238 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:53.718806 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:53.718747 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:54.937089 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:54.937050 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:55.828324 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:47:55.828290 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:48:05.422928 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:48:05.422848 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:48:36.152748 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:48:36.152715 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:48:37.882067 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:48:37.882030 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:48:52.276418 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:48:52.276369 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:48:52.365650 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:48:52.365615 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:48.950494 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:48.950414 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:49.025008 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.024956 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:49.108378 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.108346 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:49.434121 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.434049 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl"] Aug 04 18:49:49.434661 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.434642 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerName="istio-proxy" Aug 04 18:49:49.434745 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.434664 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerName="istio-proxy" Aug 04 18:49:49.434745 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.434678 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerName="istio-proxy" Aug 04 18:49:49.434745 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.434688 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerName="istio-proxy" Aug 04 18:49:49.434909 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.434821 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerName="istio-proxy" Aug 04 18:49:49.434909 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.434832 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f2bbed-9c0d-4525-8329-15a5a59cefbb" containerName="istio-proxy" Aug 04 18:49:49.438247 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.438222 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.440899 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.440878 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fee88ca8-openshift-default-dockercfg-448n9\"" Aug 04 18:49:49.458476 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.458449 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl"] Aug 04 18:49:49.464010 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.463990 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:49.576040 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.576001 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-istio-envoy\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.576195 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.576048 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-workload-socket\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.576195 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.576104 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44fa397-74f4-4117-b157-7d424804601f-istio-podinfo\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.576195 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.576143 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-workload-certs\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.576195 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.576162 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44fa397-74f4-4117-b157-7d424804601f-istio-token\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.576195 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.576180 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-istio-data\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.576471 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.576293 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb6gx\" (UniqueName: \"kubernetes.io/projected/b44fa397-74f4-4117-b157-7d424804601f-kube-api-access-qb6gx\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.576471 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.576353 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44fa397-74f4-4117-b157-7d424804601f-istiod-ca-cert\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.576471 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.576391 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-credential-socket\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.677237 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.677202 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44fa397-74f4-4117-b157-7d424804601f-istiod-ca-cert\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.677398 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.677251 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-credential-socket\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.677398 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.677302 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-istio-envoy\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.677398 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.677334 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-workload-socket\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.677398 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.677383 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44fa397-74f4-4117-b157-7d424804601f-istio-podinfo\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.677605 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.677434 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-workload-certs\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.677605 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.677463 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44fa397-74f4-4117-b157-7d424804601f-istio-token\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.677605 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.677490 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-istio-data\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.677605 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.677525 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb6gx\" (UniqueName: \"kubernetes.io/projected/b44fa397-74f4-4117-b157-7d424804601f-kube-api-access-qb6gx\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.677802 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.677744 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-credential-socket\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.677860 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.677800 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-workload-certs\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.678000 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.677957 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44fa397-74f4-4117-b157-7d424804601f-istiod-ca-cert\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.678114 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.678093 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-istio-data\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.678177 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.678135 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-workload-socket\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.679741 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.679714 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-istio-envoy\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.679865 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.679846 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44fa397-74f4-4117-b157-7d424804601f-istio-podinfo\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.687757 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.687680 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb6gx\" (UniqueName: \"kubernetes.io/projected/b44fa397-74f4-4117-b157-7d424804601f-kube-api-access-qb6gx\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.688060 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.688040 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44fa397-74f4-4117-b157-7d424804601f-istio-token\") pod \"e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.752747 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.752723 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:49.906510 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.906482 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl"] Aug 04 18:49:49.907258 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:49:49.907232 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44fa397_74f4_4117_b157_7d424804601f.slice/crio-e12d5fcc801d5afd9db662c445eb339a817963710f780f2c6b9626d980b60ea2 WatchSource:0}: Error finding container e12d5fcc801d5afd9db662c445eb339a817963710f780f2c6b9626d980b60ea2: Status 404 returned error can't find the container with id e12d5fcc801d5afd9db662c445eb339a817963710f780f2c6b9626d980b60ea2 Aug 04 18:49:49.909507 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.909476 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:49:49.909665 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.909535 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:49:49.909665 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.909584 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:49:49.963679 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:49.963655 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:50.086535 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:50.086500 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" event={"ID":"b44fa397-74f4-4117-b157-7d424804601f","Type":"ContainerStarted","Data":"d8e3cdb814e4203f4a0146ecaea26cfcd575e75610e62b5f169750b4fd5ae55a"} Aug 04 18:49:50.086703 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:50.086558 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" event={"ID":"b44fa397-74f4-4117-b157-7d424804601f","Type":"ContainerStarted","Data":"e12d5fcc801d5afd9db662c445eb339a817963710f780f2c6b9626d980b60ea2"} Aug 04 18:49:50.112436 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:50.112391 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" podStartSLOduration=1.112377815 podStartE2EDuration="1.112377815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:49:50.110534735 +0000 UTC m=+2347.361817283" watchObservedRunningTime="2026-08-04 18:49:50.112377815 +0000 UTC m=+2347.363660361" Aug 04 18:49:50.753337 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:50.753295 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:50.758176 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:50.758151 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:51.091765 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:51.091682 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:51.092701 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:51.092680 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:49:52.293802 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:52.293764 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:52.413400 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:52.413373 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:52.547265 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:52.547149 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:53.616564 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:53.616525 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:53.713624 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:53.713595 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:54.259619 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:54.259566 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:54.363711 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:49:54.363672 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:50:07.133127 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:07.133086 2526 patch_prober.go:28] interesting pod/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Aug 04 18:50:07.133504 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:07.133155 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" podUID="b44fa397-74f4-4117-b157-7d424804601f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 04 18:50:08.186508 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:08.186476 2526 generic.go:356] "Generic (PLEG): container finished" podID="b44fa397-74f4-4117-b157-7d424804601f" containerID="d8e3cdb814e4203f4a0146ecaea26cfcd575e75610e62b5f169750b4fd5ae55a" exitCode=0 Aug 04 18:50:08.186922 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:08.186547 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" event={"ID":"b44fa397-74f4-4117-b157-7d424804601f","Type":"ContainerDied","Data":"d8e3cdb814e4203f4a0146ecaea26cfcd575e75610e62b5f169750b4fd5ae55a"} Aug 04 18:50:08.186982 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:08.186955 2526 scope.go:117] "RemoveContainer" containerID="d8e3cdb814e4203f4a0146ecaea26cfcd575e75610e62b5f169750b4fd5ae55a" Aug 04 18:50:08.187957 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:08.187931 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:50:08.188032 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:08.188014 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:50:08.188071 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:08.188056 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:50:09.192432 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:09.192395 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" event={"ID":"b44fa397-74f4-4117-b157-7d424804601f","Type":"ContainerStarted","Data":"8e7b78ef0a502b89755b93d19b7e3e76dfa1bd5723ab2f7914dc1ad34429831c"} Aug 04 18:50:09.761990 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:09.761382 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:50:09.770120 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:09.770077 2526 patch_prober.go:28] interesting pod/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 04 18:50:09.770280 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:09.770148 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" podUID="b44fa397-74f4-4117-b157-7d424804601f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 04 18:50:10.753761 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:10.753720 2526 patch_prober.go:28] interesting pod/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 04 18:50:10.754418 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:10.754377 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" podUID="b44fa397-74f4-4117-b157-7d424804601f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 04 18:50:11.753767 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:11.753725 2526 patch_prober.go:28] interesting pod/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 04 18:50:11.754796 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:11.754378 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" podUID="b44fa397-74f4-4117-b157-7d424804601f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 04 18:50:12.754095 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:12.754058 2526 patch_prober.go:28] interesting pod/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 04 18:50:12.754680 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:12.754649 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" podUID="b44fa397-74f4-4117-b157-7d424804601f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 04 18:50:13.753880 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:13.753843 2526 patch_prober.go:28] interesting pod/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 04 18:50:13.754217 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:13.754187 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" podUID="b44fa397-74f4-4117-b157-7d424804601f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 04 18:50:14.756544 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:14.756513 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:50:14.756938 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:14.756860 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:50:14.757486 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:14.757470 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:50:22.469501 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:22.469455 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:50:33.866168 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:50:33.866131 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:18.035315 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:51:18.035278 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:18.192105 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:51:18.192071 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:18.301121 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:51:18.301036 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:21.717618 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:51:21.717576 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:26.865705 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:51:26.865670 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:03.365672 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:03.365640 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:04.419066 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:04.419029 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:10.432681 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:10.432646 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:10.510720 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:10.510636 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:10.702085 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:10.702049 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl"] Aug 04 18:52:10.702269 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:10.702249 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" podUID="b44fa397-74f4-4117-b157-7d424804601f" containerName="istio-proxy" containerID="cri-o://8e7b78ef0a502b89755b93d19b7e3e76dfa1bd5723ab2f7914dc1ad34429831c" gracePeriod=30 Aug 04 18:52:10.726883 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:10.726837 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:10.962737 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:10.962706 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:14.759353 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:14.759316 2526 patch_prober.go:28] interesting pod/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 18:52:14.759787 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:14.759377 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" podUID="b44fa397-74f4-4117-b157-7d424804601f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:52:15.952428 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:15.952403 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:52:16.056930 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.056858 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-workload-socket\") pod \"b44fa397-74f4-4117-b157-7d424804601f\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " Aug 04 18:52:16.056930 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.056914 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-workload-certs\") pod \"b44fa397-74f4-4117-b157-7d424804601f\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " Aug 04 18:52:16.057176 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.056935 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44fa397-74f4-4117-b157-7d424804601f-istio-token\") pod \"b44fa397-74f4-4117-b157-7d424804601f\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " Aug 04 18:52:16.057176 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.056951 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb6gx\" (UniqueName: \"kubernetes.io/projected/b44fa397-74f4-4117-b157-7d424804601f-kube-api-access-qb6gx\") pod \"b44fa397-74f4-4117-b157-7d424804601f\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " Aug 04 18:52:16.057176 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.056991 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44fa397-74f4-4117-b157-7d424804601f-istiod-ca-cert\") pod \"b44fa397-74f4-4117-b157-7d424804601f\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " Aug 04 18:52:16.057176 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.057015 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-istio-envoy\") pod \"b44fa397-74f4-4117-b157-7d424804601f\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " Aug 04 18:52:16.057176 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.057031 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-credential-socket\") pod \"b44fa397-74f4-4117-b157-7d424804601f\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " Aug 04 18:52:16.057176 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.057079 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-istio-data\") pod \"b44fa397-74f4-4117-b157-7d424804601f\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " Aug 04 18:52:16.057176 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.057103 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44fa397-74f4-4117-b157-7d424804601f-istio-podinfo\") pod \"b44fa397-74f4-4117-b157-7d424804601f\" (UID: \"b44fa397-74f4-4117-b157-7d424804601f\") " Aug 04 18:52:16.057176 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.057161 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b44fa397-74f4-4117-b157-7d424804601f" (UID: "b44fa397-74f4-4117-b157-7d424804601f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:52:16.057588 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.057176 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b44fa397-74f4-4117-b157-7d424804601f" (UID: "b44fa397-74f4-4117-b157-7d424804601f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:52:16.057588 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.057418 2526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-workload-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:52:16.057588 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.057422 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44fa397-74f4-4117-b157-7d424804601f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b44fa397-74f4-4117-b157-7d424804601f" (UID: "b44fa397-74f4-4117-b157-7d424804601f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:52:16.057588 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.057438 2526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-workload-certs\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:52:16.057588 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.057559 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-istio-data" (OuterVolumeSpecName: "istio-data") pod "b44fa397-74f4-4117-b157-7d424804601f" (UID: "b44fa397-74f4-4117-b157-7d424804601f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:52:16.057867 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.057601 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b44fa397-74f4-4117-b157-7d424804601f" (UID: "b44fa397-74f4-4117-b157-7d424804601f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:52:16.059142 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.059107 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44fa397-74f4-4117-b157-7d424804601f-istio-token" (OuterVolumeSpecName: "istio-token") pod "b44fa397-74f4-4117-b157-7d424804601f" (UID: "b44fa397-74f4-4117-b157-7d424804601f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:52:16.059253 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.059164 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b44fa397-74f4-4117-b157-7d424804601f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b44fa397-74f4-4117-b157-7d424804601f" (UID: "b44fa397-74f4-4117-b157-7d424804601f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:52:16.059308 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.059277 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44fa397-74f4-4117-b157-7d424804601f-kube-api-access-qb6gx" (OuterVolumeSpecName: "kube-api-access-qb6gx") pod "b44fa397-74f4-4117-b157-7d424804601f" (UID: "b44fa397-74f4-4117-b157-7d424804601f"). InnerVolumeSpecName "kube-api-access-qb6gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:52:16.059389 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.059362 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b44fa397-74f4-4117-b157-7d424804601f" (UID: "b44fa397-74f4-4117-b157-7d424804601f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:52:16.158669 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.158646 2526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-istio-data\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:52:16.158669 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.158666 2526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44fa397-74f4-4117-b157-7d424804601f-istio-podinfo\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:52:16.158811 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.158676 2526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44fa397-74f4-4117-b157-7d424804601f-istio-token\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:52:16.158811 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.158685 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb6gx\" (UniqueName: \"kubernetes.io/projected/b44fa397-74f4-4117-b157-7d424804601f-kube-api-access-qb6gx\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:52:16.158811 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.158694 2526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44fa397-74f4-4117-b157-7d424804601f-istiod-ca-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:52:16.158811 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.158703 2526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-istio-envoy\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:52:16.158811 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.158711 2526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44fa397-74f4-4117-b157-7d424804601f-credential-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:52:16.816856 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.816825 2526 generic.go:356] "Generic (PLEG): container finished" podID="b44fa397-74f4-4117-b157-7d424804601f" containerID="8e7b78ef0a502b89755b93d19b7e3e76dfa1bd5723ab2f7914dc1ad34429831c" exitCode=0 Aug 04 18:52:16.817059 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.816871 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" Aug 04 18:52:16.817059 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.816914 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" event={"ID":"b44fa397-74f4-4117-b157-7d424804601f","Type":"ContainerDied","Data":"8e7b78ef0a502b89755b93d19b7e3e76dfa1bd5723ab2f7914dc1ad34429831c"} Aug 04 18:52:16.817059 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.816989 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl" event={"ID":"b44fa397-74f4-4117-b157-7d424804601f","Type":"ContainerDied","Data":"e12d5fcc801d5afd9db662c445eb339a817963710f780f2c6b9626d980b60ea2"} Aug 04 18:52:16.817059 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.817013 2526 scope.go:117] "RemoveContainer" containerID="8e7b78ef0a502b89755b93d19b7e3e76dfa1bd5723ab2f7914dc1ad34429831c" Aug 04 18:52:16.826316 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.826186 2526 scope.go:117] "RemoveContainer" containerID="d8e3cdb814e4203f4a0146ecaea26cfcd575e75610e62b5f169750b4fd5ae55a" Aug 04 18:52:16.834147 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.834131 2526 scope.go:117] "RemoveContainer" containerID="8e7b78ef0a502b89755b93d19b7e3e76dfa1bd5723ab2f7914dc1ad34429831c" Aug 04 18:52:16.834375 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:52:16.834356 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7b78ef0a502b89755b93d19b7e3e76dfa1bd5723ab2f7914dc1ad34429831c\": container with ID starting with 8e7b78ef0a502b89755b93d19b7e3e76dfa1bd5723ab2f7914dc1ad34429831c not found: ID does not exist" containerID="8e7b78ef0a502b89755b93d19b7e3e76dfa1bd5723ab2f7914dc1ad34429831c" Aug 04 18:52:16.834425 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.834381 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7b78ef0a502b89755b93d19b7e3e76dfa1bd5723ab2f7914dc1ad34429831c"} err="failed to get container status \"8e7b78ef0a502b89755b93d19b7e3e76dfa1bd5723ab2f7914dc1ad34429831c\": rpc error: code = NotFound desc = could not find container \"8e7b78ef0a502b89755b93d19b7e3e76dfa1bd5723ab2f7914dc1ad34429831c\": container with ID starting with 8e7b78ef0a502b89755b93d19b7e3e76dfa1bd5723ab2f7914dc1ad34429831c not found: ID does not exist" Aug 04 18:52:16.834425 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.834396 2526 scope.go:117] "RemoveContainer" containerID="d8e3cdb814e4203f4a0146ecaea26cfcd575e75610e62b5f169750b4fd5ae55a" Aug 04 18:52:16.834616 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:52:16.834598 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e3cdb814e4203f4a0146ecaea26cfcd575e75610e62b5f169750b4fd5ae55a\": container with ID starting with d8e3cdb814e4203f4a0146ecaea26cfcd575e75610e62b5f169750b4fd5ae55a not found: ID does not exist" containerID="d8e3cdb814e4203f4a0146ecaea26cfcd575e75610e62b5f169750b4fd5ae55a" Aug 04 18:52:16.834654 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.834623 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e3cdb814e4203f4a0146ecaea26cfcd575e75610e62b5f169750b4fd5ae55a"} err="failed to get container status \"d8e3cdb814e4203f4a0146ecaea26cfcd575e75610e62b5f169750b4fd5ae55a\": rpc error: code = NotFound desc = could not find container \"d8e3cdb814e4203f4a0146ecaea26cfcd575e75610e62b5f169750b4fd5ae55a\": container with ID starting with d8e3cdb814e4203f4a0146ecaea26cfcd575e75610e62b5f169750b4fd5ae55a not found: ID does not exist" Aug 04 18:52:16.850207 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.850188 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl"] Aug 04 18:52:16.854978 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:16.854942 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fee88ca8-openshift-default-7cd44b6c44-m2xdl"] Aug 04 18:52:17.182468 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:17.182390 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b44fa397-74f4-4117-b157-7d424804601f" path="/var/lib/kubelet/pods/b44fa397-74f4-4117-b157-7d424804601f/volumes" Aug 04 18:52:22.386318 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:22.386280 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:22.548552 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:22.548514 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:22.691843 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:22.691765 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:23.258793 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:23.258760 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:23.365701 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:23.365668 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:24.802418 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:24.802389 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:32.283509 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:32.283477 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:32.387554 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:32.387526 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:33.232951 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:33.232917 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:39.671646 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:39.671559 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:39.737217 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:39.737187 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:39.832055 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:39.832030 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:39.932254 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:39.932167 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:50.426004 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:50.425953 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:50.671490 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:52:50.671462 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:11.709607 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:11.709564 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:18.526782 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:18.526737 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:18.628404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:18.628371 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:18.896386 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:18.896309 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:19.210994 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.210954 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs"] Aug 04 18:53:19.211328 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.211316 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44fa397-74f4-4117-b157-7d424804601f" containerName="istio-proxy" Aug 04 18:53:19.211393 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.211329 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44fa397-74f4-4117-b157-7d424804601f" containerName="istio-proxy" Aug 04 18:53:19.211457 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.211399 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44fa397-74f4-4117-b157-7d424804601f" containerName="istio-proxy" Aug 04 18:53:19.211457 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.211408 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44fa397-74f4-4117-b157-7d424804601f" containerName="istio-proxy" Aug 04 18:53:19.214947 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.214916 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.218347 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.218316 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-890d1a-openshift-default-dockercfg-2zrxm\"" Aug 04 18:53:19.235502 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.235475 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs"] Aug 04 18:53:19.241662 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.241637 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:19.397382 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.397352 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-459sw\" (UniqueName: \"kubernetes.io/projected/263301f8-4a10-4db0-b3a4-690b6850621c-kube-api-access-459sw\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.397549 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.397410 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263301f8-4a10-4db0-b3a4-690b6850621c-istio-podinfo\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.397549 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.397493 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-istio-envoy\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.397549 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.397528 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263301f8-4a10-4db0-b3a4-690b6850621c-istiod-ca-cert\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.397657 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.397555 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263301f8-4a10-4db0-b3a4-690b6850621c-istio-token\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.397657 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.397609 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-istio-data\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.397657 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.397632 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-workload-socket\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.397657 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.397654 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-workload-certs\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.397784 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.397674 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-credential-socket\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.498817 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.498746 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-istio-envoy\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.498817 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.498780 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263301f8-4a10-4db0-b3a4-690b6850621c-istiod-ca-cert\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.498817 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.498798 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263301f8-4a10-4db0-b3a4-690b6850621c-istio-token\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.499096 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.498821 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-istio-data\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.499096 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.498946 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-workload-socket\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.499096 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.499020 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-workload-certs\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.499096 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.499059 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-credential-socket\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.499096 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.499094 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-459sw\" (UniqueName: \"kubernetes.io/projected/263301f8-4a10-4db0-b3a4-690b6850621c-kube-api-access-459sw\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.499364 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.499202 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263301f8-4a10-4db0-b3a4-690b6850621c-istio-podinfo\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.499364 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.499322 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-istio-data\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.499558 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.499530 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263301f8-4a10-4db0-b3a4-690b6850621c-istiod-ca-cert\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.499692 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.499601 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-workload-certs\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.499692 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.499622 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-credential-socket\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.499822 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.499711 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-workload-socket\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.501021 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.500997 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-istio-envoy\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.501469 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.501452 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263301f8-4a10-4db0-b3a4-690b6850621c-istio-podinfo\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.507460 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.507433 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-459sw\" (UniqueName: \"kubernetes.io/projected/263301f8-4a10-4db0-b3a4-690b6850621c-kube-api-access-459sw\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.507460 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.507452 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263301f8-4a10-4db0-b3a4-690b6850621c-istio-token\") pod \"e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.527757 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.527737 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:19.674377 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.674338 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs"] Aug 04 18:53:19.675112 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:53:19.675088 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263301f8_4a10_4db0_b3a4_690b6850621c.slice/crio-41d00f38d5a7c3dc0b9139d0e9de4051f26c9522518c8b97671cca64de1895ba WatchSource:0}: Error finding container 41d00f38d5a7c3dc0b9139d0e9de4051f26c9522518c8b97671cca64de1895ba: Status 404 returned error can't find the container with id 41d00f38d5a7c3dc0b9139d0e9de4051f26c9522518c8b97671cca64de1895ba Aug 04 18:53:19.677555 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.677521 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:53:19.677647 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.677584 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:53:19.677647 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:19.677613 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:53:20.009635 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:20.009597 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:20.127737 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:20.127697 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" event={"ID":"263301f8-4a10-4db0-b3a4-690b6850621c","Type":"ContainerStarted","Data":"3b742051886b6649faa9d74ced17961250c4f83bdcf6ea684eb39553e8d3b3af"} Aug 04 18:53:20.127895 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:20.127761 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" event={"ID":"263301f8-4a10-4db0-b3a4-690b6850621c","Type":"ContainerStarted","Data":"41d00f38d5a7c3dc0b9139d0e9de4051f26c9522518c8b97671cca64de1895ba"} Aug 04 18:53:20.152933 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:20.152889 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" podStartSLOduration=1.152876101 podStartE2EDuration="1.152876101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:53:20.151586575 +0000 UTC m=+2557.402869125" watchObservedRunningTime="2026-08-04 18:53:20.152876101 +0000 UTC m=+2557.404158647" Aug 04 18:53:20.528792 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:20.528758 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:20.530312 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:20.530288 2526 patch_prober.go:28] interesting pod/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 18:53:20.530434 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:20.530333 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" podUID="263301f8-4a10-4db0-b3a4-690b6850621c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 18:53:21.532918 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:21.532887 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:22.137539 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:22.137498 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:22.138390 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:22.138371 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:22.200746 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:22.200672 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:22.290720 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:22.290690 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:23.448150 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:23.448117 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:23.705288 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:23.705258 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:24.047580 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.047495 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:24.143166 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.143137 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:24.303636 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.303553 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt"] Aug 04 18:53:24.303950 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.303937 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44fa397-74f4-4117-b157-7d424804601f" containerName="istio-proxy" Aug 04 18:53:24.303950 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.303950 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44fa397-74f4-4117-b157-7d424804601f" containerName="istio-proxy" Aug 04 18:53:24.307147 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.307122 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" Aug 04 18:53:24.309859 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.309835 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-890d1a-dockercfg-j2mdv\"" Aug 04 18:53:24.310259 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.310208 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-890d1a\"" Aug 04 18:53:24.318925 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.318904 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt"] Aug 04 18:53:24.342295 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.342270 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5t6p\" (UniqueName: \"kubernetes.io/projected/f7e5066e-01f8-4ca3-ace4-56980f0c1b91-kube-api-access-p5t6p\") pod \"payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt\" (UID: \"f7e5066e-01f8-4ca3-ace4-56980f0c1b91\") " pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" Aug 04 18:53:24.342422 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.342316 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e5066e-01f8-4ca3-ace4-56980f0c1b91-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt\" (UID: \"f7e5066e-01f8-4ca3-ace4-56980f0c1b91\") " pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" Aug 04 18:53:24.443569 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.443539 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5t6p\" (UniqueName: \"kubernetes.io/projected/f7e5066e-01f8-4ca3-ace4-56980f0c1b91-kube-api-access-p5t6p\") pod \"payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt\" (UID: \"f7e5066e-01f8-4ca3-ace4-56980f0c1b91\") " pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" Aug 04 18:53:24.443753 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.443583 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e5066e-01f8-4ca3-ace4-56980f0c1b91-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt\" (UID: \"f7e5066e-01f8-4ca3-ace4-56980f0c1b91\") " pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" Aug 04 18:53:24.444242 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.444220 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e5066e-01f8-4ca3-ace4-56980f0c1b91-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt\" (UID: \"f7e5066e-01f8-4ca3-ace4-56980f0c1b91\") " pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" Aug 04 18:53:24.453356 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.453334 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5t6p\" (UniqueName: \"kubernetes.io/projected/f7e5066e-01f8-4ca3-ace4-56980f0c1b91-kube-api-access-p5t6p\") pod \"payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt\" (UID: \"f7e5066e-01f8-4ca3-ace4-56980f0c1b91\") " pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" Aug 04 18:53:24.621221 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.621127 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" Aug 04 18:53:24.966505 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:24.966479 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt"] Aug 04 18:53:24.968019 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:53:24.967993 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e5066e_01f8_4ca3_ace4_56980f0c1b91.slice/crio-981b1ef8ea99385896b03f9d337ea763c43dff3208e62d72e533fb369aa4e82f WatchSource:0}: Error finding container 981b1ef8ea99385896b03f9d337ea763c43dff3208e62d72e533fb369aa4e82f: Status 404 returned error can't find the container with id 981b1ef8ea99385896b03f9d337ea763c43dff3208e62d72e533fb369aa4e82f Aug 04 18:53:25.153411 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:25.153378 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" event={"ID":"f7e5066e-01f8-4ca3-ace4-56980f0c1b91","Type":"ContainerStarted","Data":"dbd75efd6066a700b6dc5dc2bc1928a59a3f11cbd638ee2e4e8e50b50e3870ae"} Aug 04 18:53:25.153592 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:25.153437 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" event={"ID":"f7e5066e-01f8-4ca3-ace4-56980f0c1b91","Type":"ContainerStarted","Data":"981b1ef8ea99385896b03f9d337ea763c43dff3208e62d72e533fb369aa4e82f"} Aug 04 18:53:25.153592 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:25.153464 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" Aug 04 18:53:25.176197 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:25.176143 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" podStartSLOduration=1.176125044 podStartE2EDuration="1.176125044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:53:25.175446107 +0000 UTC m=+2562.426728659" watchObservedRunningTime="2026-08-04 18:53:25.176125044 +0000 UTC m=+2562.427407593" Aug 04 18:53:29.198697 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:29.198667 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:36.160176 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:36.160142 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" Aug 04 18:53:37.138764 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:37.138731 2526 patch_prober.go:28] interesting pod/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 18:53:37.139000 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:37.138801 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" podUID="263301f8-4a10-4db0-b3a4-690b6850621c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 18:53:37.215038 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:37.215003 2526 generic.go:356] "Generic (PLEG): container finished" podID="263301f8-4a10-4db0-b3a4-690b6850621c" containerID="3b742051886b6649faa9d74ced17961250c4f83bdcf6ea684eb39553e8d3b3af" exitCode=0 Aug 04 18:53:37.215418 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:37.215081 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" event={"ID":"263301f8-4a10-4db0-b3a4-690b6850621c","Type":"ContainerDied","Data":"3b742051886b6649faa9d74ced17961250c4f83bdcf6ea684eb39553e8d3b3af"} Aug 04 18:53:37.215505 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:37.215490 2526 scope.go:117] "RemoveContainer" containerID="3b742051886b6649faa9d74ced17961250c4f83bdcf6ea684eb39553e8d3b3af" Aug 04 18:53:37.216552 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:37.216524 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:53:37.216661 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:37.216606 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:53:37.216729 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:37.216656 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:53:38.222110 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:38.222076 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" event={"ID":"263301f8-4a10-4db0-b3a4-690b6850621c","Type":"ContainerStarted","Data":"7f93ff0be8ecf4e40e6810e56f0238b533291ef32aac19c4159ce3de96b1ec35"} Aug 04 18:53:38.528472 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:38.528376 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:38.528626 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:38.528607 2526 patch_prober.go:28] interesting pod/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 18:53:38.528686 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:38.528664 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" podUID="263301f8-4a10-4db0-b3a4-690b6850621c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 18:53:39.528279 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:39.528242 2526 patch_prober.go:28] interesting pod/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 18:53:39.528706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:39.528297 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" podUID="263301f8-4a10-4db0-b3a4-690b6850621c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 18:53:40.528685 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:40.528644 2526 patch_prober.go:28] interesting pod/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 18:53:40.529140 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:40.528720 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" podUID="263301f8-4a10-4db0-b3a4-690b6850621c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 18:53:41.528283 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:41.528250 2526 patch_prober.go:28] interesting pod/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 18:53:41.528468 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:41.528305 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" podUID="263301f8-4a10-4db0-b3a4-690b6850621c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 18:53:42.528106 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:42.528071 2526 patch_prober.go:28] interesting pod/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 18:53:42.528521 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:42.528126 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" podUID="263301f8-4a10-4db0-b3a4-690b6850621c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 18:53:43.532268 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:43.532235 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:43.532715 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:43.532550 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:43.533321 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:43.533304 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:53:52.567597 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:52.567558 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:52.761003 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:52.760954 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:53.112318 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.112281 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn"] Aug 04 18:53:53.116336 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.116316 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.121561 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.121545 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-716d7c-openshift-default-dockercfg-pvt8j\"" Aug 04 18:53:53.137854 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.137824 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn"] Aug 04 18:53:53.146277 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.146248 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:53.197050 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.197017 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcczc\" (UniqueName: \"kubernetes.io/projected/788ce345-6bc3-45fb-b42f-67b8361a3f90-kube-api-access-vcczc\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.197239 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.197086 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-workload-certs\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.197239 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.197186 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-envoy\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.197239 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.197231 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788ce345-6bc3-45fb-b42f-67b8361a3f90-istiod-ca-cert\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.197424 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.197311 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-data\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.197424 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.197345 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-credential-socket\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.197424 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.197382 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-token\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.197424 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.197412 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-podinfo\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.197724 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.197449 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-workload-socket\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.298118 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.298084 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788ce345-6bc3-45fb-b42f-67b8361a3f90-istiod-ca-cert\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.298293 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.298173 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-data\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.298293 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.298207 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-credential-socket\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.298293 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.298242 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-token\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.298293 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.298285 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-podinfo\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.298539 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.298328 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-workload-socket\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.298539 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.298364 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcczc\" (UniqueName: \"kubernetes.io/projected/788ce345-6bc3-45fb-b42f-67b8361a3f90-kube-api-access-vcczc\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.298539 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.298402 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-workload-certs\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.298539 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.298433 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-envoy\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.298752 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.298590 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-data\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.298810 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.298763 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-workload-socket\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.298921 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.298856 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788ce345-6bc3-45fb-b42f-67b8361a3f90-istiod-ca-cert\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.298921 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.298859 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-workload-certs\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.299077 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.299058 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-credential-socket\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.300820 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.300791 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-podinfo\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.300941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.300907 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-envoy\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.308303 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.308274 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-token\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.308523 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.308501 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcczc\" (UniqueName: \"kubernetes.io/projected/788ce345-6bc3-45fb-b42f-67b8361a3f90-kube-api-access-vcczc\") pod \"e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.427060 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.426959 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:53.575396 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.575373 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn"] Aug 04 18:53:53.577697 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:53:53.577662 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788ce345_6bc3_45fb_b42f_67b8361a3f90.slice/crio-3f2fae16bd899888fb6a01675c6a9cc070616194c00f81fa51b9284f89227c18 WatchSource:0}: Error finding container 3f2fae16bd899888fb6a01675c6a9cc070616194c00f81fa51b9284f89227c18: Status 404 returned error can't find the container with id 3f2fae16bd899888fb6a01675c6a9cc070616194c00f81fa51b9284f89227c18 Aug 04 18:53:53.579931 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.579894 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:53:53.580063 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.580010 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:53:53.580063 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.580055 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:53:53.759413 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:53.759376 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:54.305395 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:54.305355 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" event={"ID":"788ce345-6bc3-45fb-b42f-67b8361a3f90","Type":"ContainerStarted","Data":"6dcf02358be8053250fa1303154fdcb703f5547553a8ef8dc22bae5636011728"} Aug 04 18:53:54.305557 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:54.305422 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" event={"ID":"788ce345-6bc3-45fb-b42f-67b8361a3f90","Type":"ContainerStarted","Data":"3f2fae16bd899888fb6a01675c6a9cc070616194c00f81fa51b9284f89227c18"} Aug 04 18:53:54.352872 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:54.352823 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" podStartSLOduration=1.3528075099999999 podStartE2EDuration="1.35280751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:53:54.350622335 +0000 UTC m=+2591.601904873" watchObservedRunningTime="2026-08-04 18:53:54.35280751 +0000 UTC m=+2591.604090057" Aug 04 18:53:54.427536 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:54.427503 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:54.433056 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:54.433036 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:55.310912 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:55.310879 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:55.311732 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:55.311719 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:53:56.371278 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:56.371246 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:56.471108 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:56.471068 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:57.538547 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:57.538508 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:57.773689 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:57.773658 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:58.098079 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:58.098045 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:53:58.190041 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:53:58.190013 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:54:00.106476 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:00.106443 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:54:10.354353 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:10.354304 2526 patch_prober.go:28] interesting pod/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:51530->10.133.0.57:15021: read: connection reset by peer" start-of-body= Aug 04 18:54:10.354901 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:10.354389 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" podUID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:51530->10.133.0.57:15021: read: connection reset by peer" Aug 04 18:54:11.404934 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:11.404904 2526 generic.go:356] "Generic (PLEG): container finished" podID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerID="6dcf02358be8053250fa1303154fdcb703f5547553a8ef8dc22bae5636011728" exitCode=0 Aug 04 18:54:11.405345 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:11.404986 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" event={"ID":"788ce345-6bc3-45fb-b42f-67b8361a3f90","Type":"ContainerDied","Data":"6dcf02358be8053250fa1303154fdcb703f5547553a8ef8dc22bae5636011728"} Aug 04 18:54:11.405345 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:11.405328 2526 scope.go:117] "RemoveContainer" containerID="6dcf02358be8053250fa1303154fdcb703f5547553a8ef8dc22bae5636011728" Aug 04 18:54:11.406284 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:11.406258 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:54:11.406389 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:11.406321 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:54:11.406389 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:11.406354 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:54:12.412047 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:12.412012 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" event={"ID":"788ce345-6bc3-45fb-b42f-67b8361a3f90","Type":"ContainerStarted","Data":"5844d057fefa43473ac128262061d35f74bcde9d9926f5cffd74192dd021eb15"} Aug 04 18:54:12.427380 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:12.427354 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:54:12.427556 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:12.427537 2526 patch_prober.go:28] interesting pod/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 18:54:12.427634 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:12.427589 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" podUID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 18:54:13.427792 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:13.427755 2526 patch_prober.go:28] interesting pod/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 18:54:13.428201 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:13.427803 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" podUID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 18:54:14.427430 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:14.427391 2526 patch_prober.go:28] interesting pod/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 18:54:14.427634 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:14.427449 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" podUID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 18:54:15.427807 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:15.427769 2526 patch_prober.go:28] interesting pod/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 18:54:15.428307 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:15.427826 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" podUID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 18:54:16.427631 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:16.427588 2526 patch_prober.go:28] interesting pod/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 18:54:16.427822 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:16.427648 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" podUID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 18:54:17.428264 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:17.428226 2526 patch_prober.go:28] interesting pod/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 18:54:17.428656 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:17.428281 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" podUID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 18:54:18.431013 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:18.430982 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:54:18.431428 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:18.431330 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:54:18.432066 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:18.432049 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:54:21.503657 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:21.503623 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:54:36.238857 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:54:36.238826 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:55:00.857584 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:00.857544 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:55:31.843135 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:31.843102 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:55:41.802912 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:41.802826 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:55:42.716917 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:42.716883 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:55:47.267394 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.267357 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:55:47.324468 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.324429 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt"] Aug 04 18:55:47.324744 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.324714 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" podUID="f7e5066e-01f8-4ca3-ace4-56980f0c1b91" containerName="payload-processing" containerID="cri-o://dbd75efd6066a700b6dc5dc2bc1928a59a3f11cbd638ee2e4e8e50b50e3870ae" gracePeriod=30 Aug 04 18:55:47.390942 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.390898 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:55:47.491370 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.491329 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:55:47.566658 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.566625 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:55:47.582628 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.582607 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" Aug 04 18:55:47.720844 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.720806 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5t6p\" (UniqueName: \"kubernetes.io/projected/f7e5066e-01f8-4ca3-ace4-56980f0c1b91-kube-api-access-p5t6p\") pod \"f7e5066e-01f8-4ca3-ace4-56980f0c1b91\" (UID: \"f7e5066e-01f8-4ca3-ace4-56980f0c1b91\") " Aug 04 18:55:47.721055 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.720884 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e5066e-01f8-4ca3-ace4-56980f0c1b91-plugins-config-volume\") pod \"f7e5066e-01f8-4ca3-ace4-56980f0c1b91\" (UID: \"f7e5066e-01f8-4ca3-ace4-56980f0c1b91\") " Aug 04 18:55:47.721274 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.721247 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e5066e-01f8-4ca3-ace4-56980f0c1b91-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7e5066e-01f8-4ca3-ace4-56980f0c1b91" (UID: "f7e5066e-01f8-4ca3-ace4-56980f0c1b91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:55:47.722829 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.722808 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e5066e-01f8-4ca3-ace4-56980f0c1b91-kube-api-access-p5t6p" (OuterVolumeSpecName: "kube-api-access-p5t6p") pod "f7e5066e-01f8-4ca3-ace4-56980f0c1b91" (UID: "f7e5066e-01f8-4ca3-ace4-56980f0c1b91"). InnerVolumeSpecName "kube-api-access-p5t6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:55:47.822306 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.822233 2526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e5066e-01f8-4ca3-ace4-56980f0c1b91-plugins-config-volume\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:55:47.822306 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.822258 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5t6p\" (UniqueName: \"kubernetes.io/projected/f7e5066e-01f8-4ca3-ace4-56980f0c1b91-kube-api-access-p5t6p\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:55:47.887507 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.887470 2526 generic.go:356] "Generic (PLEG): container finished" podID="f7e5066e-01f8-4ca3-ace4-56980f0c1b91" containerID="dbd75efd6066a700b6dc5dc2bc1928a59a3f11cbd638ee2e4e8e50b50e3870ae" exitCode=0 Aug 04 18:55:47.887678 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.887518 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" Aug 04 18:55:47.887678 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.887549 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" event={"ID":"f7e5066e-01f8-4ca3-ace4-56980f0c1b91","Type":"ContainerDied","Data":"dbd75efd6066a700b6dc5dc2bc1928a59a3f11cbd638ee2e4e8e50b50e3870ae"} Aug 04 18:55:47.887678 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.887598 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt" event={"ID":"f7e5066e-01f8-4ca3-ace4-56980f0c1b91","Type":"ContainerDied","Data":"981b1ef8ea99385896b03f9d337ea763c43dff3208e62d72e533fb369aa4e82f"} Aug 04 18:55:47.887678 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.887619 2526 scope.go:117] "RemoveContainer" containerID="dbd75efd6066a700b6dc5dc2bc1928a59a3f11cbd638ee2e4e8e50b50e3870ae" Aug 04 18:55:47.897316 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.897213 2526 scope.go:117] "RemoveContainer" containerID="dbd75efd6066a700b6dc5dc2bc1928a59a3f11cbd638ee2e4e8e50b50e3870ae" Aug 04 18:55:47.897570 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:55:47.897546 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd75efd6066a700b6dc5dc2bc1928a59a3f11cbd638ee2e4e8e50b50e3870ae\": container with ID starting with dbd75efd6066a700b6dc5dc2bc1928a59a3f11cbd638ee2e4e8e50b50e3870ae not found: ID does not exist" containerID="dbd75efd6066a700b6dc5dc2bc1928a59a3f11cbd638ee2e4e8e50b50e3870ae" Aug 04 18:55:47.897659 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.897582 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd75efd6066a700b6dc5dc2bc1928a59a3f11cbd638ee2e4e8e50b50e3870ae"} err="failed to get container status \"dbd75efd6066a700b6dc5dc2bc1928a59a3f11cbd638ee2e4e8e50b50e3870ae\": rpc error: code = NotFound desc = could not find container \"dbd75efd6066a700b6dc5dc2bc1928a59a3f11cbd638ee2e4e8e50b50e3870ae\": container with ID starting with dbd75efd6066a700b6dc5dc2bc1928a59a3f11cbd638ee2e4e8e50b50e3870ae not found: ID does not exist" Aug 04 18:55:47.919565 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.919535 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt"] Aug 04 18:55:47.931440 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:47.931416 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-890d1a-6594bd9fbd-2bgwt"] Aug 04 18:55:48.070181 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:48.070151 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:55:49.181405 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:49.181371 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e5066e-01f8-4ca3-ace4-56980f0c1b91" path="/var/lib/kubelet/pods/f7e5066e-01f8-4ca3-ace4-56980f0c1b91/volumes" Aug 04 18:55:53.023808 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:53.023775 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:55:59.832012 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:55:59.831957 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:00.096999 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:00.096879 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs"] Aug 04 18:56:00.097223 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:00.097184 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" podUID="263301f8-4a10-4db0-b3a4-690b6850621c" containerName="istio-proxy" containerID="cri-o://7f93ff0be8ecf4e40e6810e56f0238b533291ef32aac19c4159ce3de96b1ec35" gracePeriod=30 Aug 04 18:56:00.120312 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:00.120282 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:00.356873 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:00.356781 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:05.353769 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.353745 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:56:05.474648 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.474626 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-459sw\" (UniqueName: \"kubernetes.io/projected/263301f8-4a10-4db0-b3a4-690b6850621c-kube-api-access-459sw\") pod \"263301f8-4a10-4db0-b3a4-690b6850621c\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " Aug 04 18:56:05.474823 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.474655 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263301f8-4a10-4db0-b3a4-690b6850621c-istio-podinfo\") pod \"263301f8-4a10-4db0-b3a4-690b6850621c\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " Aug 04 18:56:05.474823 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.474701 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-istio-envoy\") pod \"263301f8-4a10-4db0-b3a4-690b6850621c\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " Aug 04 18:56:05.474823 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.474726 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263301f8-4a10-4db0-b3a4-690b6850621c-istiod-ca-cert\") pod \"263301f8-4a10-4db0-b3a4-690b6850621c\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " Aug 04 18:56:05.474823 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.474778 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263301f8-4a10-4db0-b3a4-690b6850621c-istio-token\") pod \"263301f8-4a10-4db0-b3a4-690b6850621c\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " Aug 04 18:56:05.474823 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.474801 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-credential-socket\") pod \"263301f8-4a10-4db0-b3a4-690b6850621c\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " Aug 04 18:56:05.475120 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.474826 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-istio-data\") pod \"263301f8-4a10-4db0-b3a4-690b6850621c\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " Aug 04 18:56:05.475120 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.474858 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-workload-certs\") pod \"263301f8-4a10-4db0-b3a4-690b6850621c\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " Aug 04 18:56:05.475120 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.474915 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-workload-socket\") pod \"263301f8-4a10-4db0-b3a4-690b6850621c\" (UID: \"263301f8-4a10-4db0-b3a4-690b6850621c\") " Aug 04 18:56:05.475255 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.475235 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263301f8-4a10-4db0-b3a4-690b6850621c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "263301f8-4a10-4db0-b3a4-690b6850621c" (UID: "263301f8-4a10-4db0-b3a4-690b6850621c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:56:05.475374 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.475348 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-istio-data" (OuterVolumeSpecName: "istio-data") pod "263301f8-4a10-4db0-b3a4-690b6850621c" (UID: "263301f8-4a10-4db0-b3a4-690b6850621c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:56:05.475432 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.475353 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "263301f8-4a10-4db0-b3a4-690b6850621c" (UID: "263301f8-4a10-4db0-b3a4-690b6850621c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:56:05.475484 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.475414 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "263301f8-4a10-4db0-b3a4-690b6850621c" (UID: "263301f8-4a10-4db0-b3a4-690b6850621c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:56:05.475484 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.475474 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "263301f8-4a10-4db0-b3a4-690b6850621c" (UID: "263301f8-4a10-4db0-b3a4-690b6850621c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:56:05.477092 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.477053 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263301f8-4a10-4db0-b3a4-690b6850621c-istio-token" (OuterVolumeSpecName: "istio-token") pod "263301f8-4a10-4db0-b3a4-690b6850621c" (UID: "263301f8-4a10-4db0-b3a4-690b6850621c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:56:05.477201 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.477136 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/263301f8-4a10-4db0-b3a4-690b6850621c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "263301f8-4a10-4db0-b3a4-690b6850621c" (UID: "263301f8-4a10-4db0-b3a4-690b6850621c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:56:05.477359 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.477343 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263301f8-4a10-4db0-b3a4-690b6850621c-kube-api-access-459sw" (OuterVolumeSpecName: "kube-api-access-459sw") pod "263301f8-4a10-4db0-b3a4-690b6850621c" (UID: "263301f8-4a10-4db0-b3a4-690b6850621c"). InnerVolumeSpecName "kube-api-access-459sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:56:05.477428 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.477358 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "263301f8-4a10-4db0-b3a4-690b6850621c" (UID: "263301f8-4a10-4db0-b3a4-690b6850621c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:56:05.575848 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.575825 2526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-istio-envoy\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:05.575848 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.575847 2526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263301f8-4a10-4db0-b3a4-690b6850621c-istiod-ca-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:05.576026 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.575856 2526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263301f8-4a10-4db0-b3a4-690b6850621c-istio-token\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:05.576026 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.575864 2526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-credential-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:05.576026 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.575872 2526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-istio-data\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:05.576026 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.575882 2526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-workload-certs\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:05.576026 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.575890 2526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263301f8-4a10-4db0-b3a4-690b6850621c-workload-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:05.576026 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.575899 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-459sw\" (UniqueName: \"kubernetes.io/projected/263301f8-4a10-4db0-b3a4-690b6850621c-kube-api-access-459sw\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:05.576026 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.575907 2526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263301f8-4a10-4db0-b3a4-690b6850621c-istio-podinfo\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:05.984530 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.984498 2526 generic.go:356] "Generic (PLEG): container finished" podID="263301f8-4a10-4db0-b3a4-690b6850621c" containerID="7f93ff0be8ecf4e40e6810e56f0238b533291ef32aac19c4159ce3de96b1ec35" exitCode=0 Aug 04 18:56:05.984693 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.984549 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" Aug 04 18:56:05.984693 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.984561 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" event={"ID":"263301f8-4a10-4db0-b3a4-690b6850621c","Type":"ContainerDied","Data":"7f93ff0be8ecf4e40e6810e56f0238b533291ef32aac19c4159ce3de96b1ec35"} Aug 04 18:56:05.984693 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.984603 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs" event={"ID":"263301f8-4a10-4db0-b3a4-690b6850621c","Type":"ContainerDied","Data":"41d00f38d5a7c3dc0b9139d0e9de4051f26c9522518c8b97671cca64de1895ba"} Aug 04 18:56:05.984693 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.984625 2526 scope.go:117] "RemoveContainer" containerID="7f93ff0be8ecf4e40e6810e56f0238b533291ef32aac19c4159ce3de96b1ec35" Aug 04 18:56:05.993836 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:05.993816 2526 scope.go:117] "RemoveContainer" containerID="3b742051886b6649faa9d74ced17961250c4f83bdcf6ea684eb39553e8d3b3af" Aug 04 18:56:06.001482 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:06.001466 2526 scope.go:117] "RemoveContainer" containerID="7f93ff0be8ecf4e40e6810e56f0238b533291ef32aac19c4159ce3de96b1ec35" Aug 04 18:56:06.001738 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:56:06.001719 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f93ff0be8ecf4e40e6810e56f0238b533291ef32aac19c4159ce3de96b1ec35\": container with ID starting with 7f93ff0be8ecf4e40e6810e56f0238b533291ef32aac19c4159ce3de96b1ec35 not found: ID does not exist" containerID="7f93ff0be8ecf4e40e6810e56f0238b533291ef32aac19c4159ce3de96b1ec35" Aug 04 18:56:06.001788 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:06.001746 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f93ff0be8ecf4e40e6810e56f0238b533291ef32aac19c4159ce3de96b1ec35"} err="failed to get container status \"7f93ff0be8ecf4e40e6810e56f0238b533291ef32aac19c4159ce3de96b1ec35\": rpc error: code = NotFound desc = could not find container \"7f93ff0be8ecf4e40e6810e56f0238b533291ef32aac19c4159ce3de96b1ec35\": container with ID starting with 7f93ff0be8ecf4e40e6810e56f0238b533291ef32aac19c4159ce3de96b1ec35 not found: ID does not exist" Aug 04 18:56:06.001788 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:06.001764 2526 scope.go:117] "RemoveContainer" containerID="3b742051886b6649faa9d74ced17961250c4f83bdcf6ea684eb39553e8d3b3af" Aug 04 18:56:06.001988 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:56:06.001957 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b742051886b6649faa9d74ced17961250c4f83bdcf6ea684eb39553e8d3b3af\": container with ID starting with 3b742051886b6649faa9d74ced17961250c4f83bdcf6ea684eb39553e8d3b3af not found: ID does not exist" containerID="3b742051886b6649faa9d74ced17961250c4f83bdcf6ea684eb39553e8d3b3af" Aug 04 18:56:06.002037 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:06.001994 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b742051886b6649faa9d74ced17961250c4f83bdcf6ea684eb39553e8d3b3af"} err="failed to get container status \"3b742051886b6649faa9d74ced17961250c4f83bdcf6ea684eb39553e8d3b3af\": rpc error: code = NotFound desc = could not find container \"3b742051886b6649faa9d74ced17961250c4f83bdcf6ea684eb39553e8d3b3af\": container with ID starting with 3b742051886b6649faa9d74ced17961250c4f83bdcf6ea684eb39553e8d3b3af not found: ID does not exist" Aug 04 18:56:06.010923 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:06.010903 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs"] Aug 04 18:56:06.019643 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:06.019623 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-890d1a-openshift-default-756c7b7bdd-cwxhs"] Aug 04 18:56:07.181059 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:07.181026 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="263301f8-4a10-4db0-b3a4-690b6850621c" path="/var/lib/kubelet/pods/263301f8-4a10-4db0-b3a4-690b6850621c/volumes" Aug 04 18:56:11.202010 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:11.201959 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:11.286019 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:11.285956 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:11.360839 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:11.360805 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:11.436698 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:11.436665 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:11.915280 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:11.915245 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:18.562945 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:18.562897 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:18.646508 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:18.646480 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:18.864803 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:18.864723 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn"] Aug 04 18:56:18.865003 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:18.864955 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" podUID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerName="istio-proxy" containerID="cri-o://5844d057fefa43473ac128262061d35f74bcde9d9926f5cffd74192dd021eb15" gracePeriod=30 Aug 04 18:56:18.876529 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:18.876498 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:19.115752 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.115658 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:19.544924 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.544895 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:19.804501 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.804415 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22"] Aug 04 18:56:19.804927 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.804843 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="263301f8-4a10-4db0-b3a4-690b6850621c" containerName="istio-proxy" Aug 04 18:56:19.804927 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.804857 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="263301f8-4a10-4db0-b3a4-690b6850621c" containerName="istio-proxy" Aug 04 18:56:19.804927 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.804875 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="263301f8-4a10-4db0-b3a4-690b6850621c" containerName="istio-proxy" Aug 04 18:56:19.804927 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.804881 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="263301f8-4a10-4db0-b3a4-690b6850621c" containerName="istio-proxy" Aug 04 18:56:19.804927 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.804893 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e5066e-01f8-4ca3-ace4-56980f0c1b91" containerName="payload-processing" Aug 04 18:56:19.804927 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.804899 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e5066e-01f8-4ca3-ace4-56980f0c1b91" containerName="payload-processing" Aug 04 18:56:19.805297 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.804984 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="263301f8-4a10-4db0-b3a4-690b6850621c" containerName="istio-proxy" Aug 04 18:56:19.805297 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.804994 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="263301f8-4a10-4db0-b3a4-690b6850621c" containerName="istio-proxy" Aug 04 18:56:19.805297 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.805004 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e5066e-01f8-4ca3-ace4-56980f0c1b91" containerName="payload-processing" Aug 04 18:56:19.809705 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.809687 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:19.812608 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.812592 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-95504a-openshift-default-dockercfg-4kh82\"" Aug 04 18:56:19.821547 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.821525 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22"] Aug 04 18:56:19.829170 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.829144 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:19.900804 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.900763 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp4hj\" (UniqueName: \"kubernetes.io/projected/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-kube-api-access-fp4hj\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:19.901000 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.900815 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-podinfo\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:19.901000 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.900956 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-workload-certs\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:19.901091 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.901017 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-workload-socket\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:19.901091 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.901049 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-token\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:19.901091 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.901076 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-envoy\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:19.901209 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.901102 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-data\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:19.901209 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.901196 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-credential-socket\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:19.901288 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.901228 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istiod-ca-cert\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:19.947475 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:19.947444 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:20.002439 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.002401 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-data\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.002612 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.002533 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-credential-socket\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.002612 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.002556 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istiod-ca-cert\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.002612 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.002578 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp4hj\" (UniqueName: \"kubernetes.io/projected/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-kube-api-access-fp4hj\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.002785 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.002617 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-podinfo\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.002785 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.002701 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-workload-certs\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.002785 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.002733 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-workload-socket\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.002785 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.002762 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-token\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.003041 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.002792 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-envoy\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.003041 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.002811 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-data\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.003041 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.002909 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-credential-socket\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.003211 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.003066 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-workload-certs\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.003211 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.003128 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-workload-socket\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.003354 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.003329 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istiod-ca-cert\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.005120 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.005092 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-envoy\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.005323 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.005305 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-podinfo\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.014711 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.014686 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-token\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.014999 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.014959 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp4hj\" (UniqueName: \"kubernetes.io/projected/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-kube-api-access-fp4hj\") pod \"e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.121223 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.121147 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:20.263455 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.263430 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22"] Aug 04 18:56:20.265299 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:56:20.265271 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d780e6_9605_4e8c_aeb2_984a2f2d9bb4.slice/crio-2c032ca6caa3fe2a71340b47a8d00739ae8e23c47611e051689e0e4639a06f3e WatchSource:0}: Error finding container 2c032ca6caa3fe2a71340b47a8d00739ae8e23c47611e051689e0e4639a06f3e: Status 404 returned error can't find the container with id 2c032ca6caa3fe2a71340b47a8d00739ae8e23c47611e051689e0e4639a06f3e Aug 04 18:56:20.267518 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.267488 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:56:20.267590 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.267552 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:56:20.267630 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:20.267601 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:56:21.062900 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:21.062863 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" event={"ID":"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4","Type":"ContainerStarted","Data":"0e81df0709a1c888d62fe3b7a15ae45213505ffbcec65d036ef2f514378be768"} Aug 04 18:56:21.063290 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:21.062922 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" event={"ID":"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4","Type":"ContainerStarted","Data":"2c032ca6caa3fe2a71340b47a8d00739ae8e23c47611e051689e0e4639a06f3e"} Aug 04 18:56:21.090229 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:21.090185 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" podStartSLOduration=2.090169682 podStartE2EDuration="2.090169682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:56:21.088270593 +0000 UTC m=+2738.339553141" watchObservedRunningTime="2026-08-04 18:56:21.090169682 +0000 UTC m=+2738.341452229" Aug 04 18:56:21.122098 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:21.122073 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:21.127003 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:21.126955 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:22.068206 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:22.068174 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:22.069104 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:22.069084 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:23.103369 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:23.103329 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:24.082410 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.082378 2526 generic.go:356] "Generic (PLEG): container finished" podID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerID="5844d057fefa43473ac128262061d35f74bcde9d9926f5cffd74192dd021eb15" exitCode=0 Aug 04 18:56:24.082572 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.082448 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" event={"ID":"788ce345-6bc3-45fb-b42f-67b8361a3f90","Type":"ContainerDied","Data":"5844d057fefa43473ac128262061d35f74bcde9d9926f5cffd74192dd021eb15"} Aug 04 18:56:24.082572 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.082510 2526 scope.go:117] "RemoveContainer" containerID="6dcf02358be8053250fa1303154fdcb703f5547553a8ef8dc22bae5636011728" Aug 04 18:56:24.123104 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.123082 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:56:24.243330 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.243295 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-workload-certs\") pod \"788ce345-6bc3-45fb-b42f-67b8361a3f90\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " Aug 04 18:56:24.243516 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.243342 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcczc\" (UniqueName: \"kubernetes.io/projected/788ce345-6bc3-45fb-b42f-67b8361a3f90-kube-api-access-vcczc\") pod \"788ce345-6bc3-45fb-b42f-67b8361a3f90\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " Aug 04 18:56:24.243516 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.243361 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-credential-socket\") pod \"788ce345-6bc3-45fb-b42f-67b8361a3f90\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " Aug 04 18:56:24.243516 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.243386 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-token\") pod \"788ce345-6bc3-45fb-b42f-67b8361a3f90\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " Aug 04 18:56:24.243516 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.243413 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-podinfo\") pod \"788ce345-6bc3-45fb-b42f-67b8361a3f90\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " Aug 04 18:56:24.243516 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.243448 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-workload-socket\") pod \"788ce345-6bc3-45fb-b42f-67b8361a3f90\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " Aug 04 18:56:24.243516 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.243480 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788ce345-6bc3-45fb-b42f-67b8361a3f90-istiod-ca-cert\") pod \"788ce345-6bc3-45fb-b42f-67b8361a3f90\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " Aug 04 18:56:24.243853 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.243593 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-envoy\") pod \"788ce345-6bc3-45fb-b42f-67b8361a3f90\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " Aug 04 18:56:24.243853 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.243603 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "788ce345-6bc3-45fb-b42f-67b8361a3f90" (UID: "788ce345-6bc3-45fb-b42f-67b8361a3f90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:56:24.243853 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.243627 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-data\") pod \"788ce345-6bc3-45fb-b42f-67b8361a3f90\" (UID: \"788ce345-6bc3-45fb-b42f-67b8361a3f90\") " Aug 04 18:56:24.244037 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.243987 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-data" (OuterVolumeSpecName: "istio-data") pod "788ce345-6bc3-45fb-b42f-67b8361a3f90" (UID: "788ce345-6bc3-45fb-b42f-67b8361a3f90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:56:24.244037 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.244026 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788ce345-6bc3-45fb-b42f-67b8361a3f90-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "788ce345-6bc3-45fb-b42f-67b8361a3f90" (UID: "788ce345-6bc3-45fb-b42f-67b8361a3f90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:56:24.244214 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.244186 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "788ce345-6bc3-45fb-b42f-67b8361a3f90" (UID: "788ce345-6bc3-45fb-b42f-67b8361a3f90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:56:24.244298 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.244254 2526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-workload-certs\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:24.244398 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.244366 2526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-data\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:24.244519 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.244496 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "788ce345-6bc3-45fb-b42f-67b8361a3f90" (UID: "788ce345-6bc3-45fb-b42f-67b8361a3f90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:56:24.246080 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.246056 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788ce345-6bc3-45fb-b42f-67b8361a3f90-kube-api-access-vcczc" (OuterVolumeSpecName: "kube-api-access-vcczc") pod "788ce345-6bc3-45fb-b42f-67b8361a3f90" (UID: "788ce345-6bc3-45fb-b42f-67b8361a3f90"). InnerVolumeSpecName "kube-api-access-vcczc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:56:24.246325 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.246303 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-token" (OuterVolumeSpecName: "istio-token") pod "788ce345-6bc3-45fb-b42f-67b8361a3f90" (UID: "788ce345-6bc3-45fb-b42f-67b8361a3f90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:56:24.246557 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.246526 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "788ce345-6bc3-45fb-b42f-67b8361a3f90" (UID: "788ce345-6bc3-45fb-b42f-67b8361a3f90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:56:24.246646 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.246601 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "788ce345-6bc3-45fb-b42f-67b8361a3f90" (UID: "788ce345-6bc3-45fb-b42f-67b8361a3f90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:56:24.345658 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.345622 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcczc\" (UniqueName: \"kubernetes.io/projected/788ce345-6bc3-45fb-b42f-67b8361a3f90-kube-api-access-vcczc\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:24.345658 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.345653 2526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-credential-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:24.345658 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.345667 2526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-token\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:24.345904 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.345681 2526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-podinfo\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:24.345904 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.345695 2526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-workload-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:24.345904 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.345708 2526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788ce345-6bc3-45fb-b42f-67b8361a3f90-istiod-ca-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:24.345904 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.345722 2526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788ce345-6bc3-45fb-b42f-67b8361a3f90-istio-envoy\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:56:24.734630 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.734597 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:24.837543 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:24.837513 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:25.090598 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:25.090510 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" Aug 04 18:56:25.090831 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:25.090798 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn" event={"ID":"788ce345-6bc3-45fb-b42f-67b8361a3f90","Type":"ContainerDied","Data":"3f2fae16bd899888fb6a01675c6a9cc070616194c00f81fa51b9284f89227c18"} Aug 04 18:56:25.090988 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:25.090862 2526 scope.go:117] "RemoveContainer" containerID="5844d057fefa43473ac128262061d35f74bcde9d9926f5cffd74192dd021eb15" Aug 04 18:56:25.121670 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:25.121639 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn"] Aug 04 18:56:25.129501 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:25.129470 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-716d7c-openshift-default-759bf8c5c6-zd6rn"] Aug 04 18:56:25.181427 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:25.181386 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="788ce345-6bc3-45fb-b42f-67b8361a3f90" path="/var/lib/kubelet/pods/788ce345-6bc3-45fb-b42f-67b8361a3f90/volumes" Aug 04 18:56:25.275980 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:25.275934 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:37.070256 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:37.070187 2526 patch_prober.go:28] interesting pod/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Aug 04 18:56:37.070632 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:37.070249 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" podUID="c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Aug 04 18:56:37.160884 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:37.160815 2526 generic.go:356] "Generic (PLEG): container finished" podID="c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" containerID="0e81df0709a1c888d62fe3b7a15ae45213505ffbcec65d036ef2f514378be768" exitCode=0 Aug 04 18:56:37.161016 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:37.160886 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" event={"ID":"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4","Type":"ContainerDied","Data":"0e81df0709a1c888d62fe3b7a15ae45213505ffbcec65d036ef2f514378be768"} Aug 04 18:56:37.161283 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:37.161269 2526 scope.go:117] "RemoveContainer" containerID="0e81df0709a1c888d62fe3b7a15ae45213505ffbcec65d036ef2f514378be768" Aug 04 18:56:37.162300 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:37.162274 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:56:37.162359 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:37.162346 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:56:37.162397 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:37.162376 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:56:38.171387 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:38.171350 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" event={"ID":"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4","Type":"ContainerStarted","Data":"3aa28e4f01512a8a76f4d45e4d49d0a9f09512dd12b83985efd2d4dabac3fb3d"} Aug 04 18:56:39.122182 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:39.122153 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:39.122362 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:39.122301 2526 patch_prober.go:28] interesting pod/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Aug 04 18:56:39.122408 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:39.122355 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" podUID="c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Aug 04 18:56:40.121992 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:40.121946 2526 patch_prober.go:28] interesting pod/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Aug 04 18:56:40.122468 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:40.122012 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" podUID="c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Aug 04 18:56:41.122542 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:41.122507 2526 patch_prober.go:28] interesting pod/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Aug 04 18:56:41.122937 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:41.122561 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" podUID="c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Aug 04 18:56:42.121837 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:42.121801 2526 patch_prober.go:28] interesting pod/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Aug 04 18:56:42.122039 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:42.121857 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" podUID="c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Aug 04 18:56:43.125898 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:43.125869 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:43.126343 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:43.126205 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:43.126825 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:43.126806 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:56:48.115880 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:48.115845 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:48.671928 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:48.671892 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:48.958164 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:48.958136 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2"] Aug 04 18:56:48.958521 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:48.958509 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerName="istio-proxy" Aug 04 18:56:48.958566 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:48.958523 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerName="istio-proxy" Aug 04 18:56:48.958607 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:48.958586 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerName="istio-proxy" Aug 04 18:56:48.958607 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:48.958594 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerName="istio-proxy" Aug 04 18:56:48.961246 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:48.961227 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:48.964237 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:48.964211 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0de7a2-openshift-default-dockercfg-wdn2l\"" Aug 04 18:56:48.975902 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:48.975882 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2"] Aug 04 18:56:49.000355 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.000331 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:49.061393 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.061362 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-envoy\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.061582 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.061400 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-data\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.061582 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.061478 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istiod-ca-cert\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.061695 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.061589 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-credential-socket\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.061695 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.061641 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-token\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.061695 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.061681 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dbbm\" (UniqueName: \"kubernetes.io/projected/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-kube-api-access-4dbbm\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.061857 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.061734 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-workload-socket\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.061857 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.061772 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-podinfo\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.061857 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.061831 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-workload-certs\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.162699 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.162662 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-workload-certs\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.163146 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.162711 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-envoy\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.163146 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.162826 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-data\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.163146 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.162874 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istiod-ca-cert\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.163146 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.162956 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-credential-socket\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.163146 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.163028 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-token\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.163146 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.163063 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dbbm\" (UniqueName: \"kubernetes.io/projected/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-kube-api-access-4dbbm\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.163146 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.163074 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-workload-certs\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.163146 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.163111 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-workload-socket\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.163537 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.163156 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-podinfo\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.163537 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.163161 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-data\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.163537 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.163326 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-credential-socket\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.163537 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.163442 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-workload-socket\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.163680 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.163547 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istiod-ca-cert\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.165012 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.164991 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-envoy\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.165511 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.165495 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-podinfo\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.173727 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.173701 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-token\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.173940 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.173919 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dbbm\" (UniqueName: \"kubernetes.io/projected/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-kube-api-access-4dbbm\") pod \"e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.177749 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.177722 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:49.271776 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.271702 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:49.408361 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.408337 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2"] Aug 04 18:56:49.409937 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:56:49.409910 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4985c7a2_f2c9_4f67_a773_fb3d54a13aa9.slice/crio-e0307c56213152c1ee75eafb09b36cd8425990e9c5b71cc2241b4c8df207d4cb WatchSource:0}: Error finding container e0307c56213152c1ee75eafb09b36cd8425990e9c5b71cc2241b4c8df207d4cb: Status 404 returned error can't find the container with id e0307c56213152c1ee75eafb09b36cd8425990e9c5b71cc2241b4c8df207d4cb Aug 04 18:56:49.412530 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.412490 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:56:49.412654 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.412557 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:56:49.412654 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:49.412586 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:56:50.237181 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:50.237144 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" event={"ID":"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9","Type":"ContainerStarted","Data":"8759126f45b6ccd4db10f6c838ac4d013fcb706ca148e6fe5cc4666f02a13d73"} Aug 04 18:56:50.237589 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:50.237191 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" event={"ID":"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9","Type":"ContainerStarted","Data":"e0307c56213152c1ee75eafb09b36cd8425990e9c5b71cc2241b4c8df207d4cb"} Aug 04 18:56:50.270683 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:50.270639 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" podStartSLOduration=2.270623136 podStartE2EDuration="2.270623136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:56:50.268600616 +0000 UTC m=+2767.519883164" watchObservedRunningTime="2026-08-04 18:56:50.270623136 +0000 UTC m=+2767.521905753" Aug 04 18:56:50.271988 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:50.271890 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:50.276456 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:50.276434 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:51.242506 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:51.242470 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:51.243282 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:51.243262 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:56:52.389526 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:52.389473 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:52.482053 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:52.482022 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:53.583014 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:53.582987 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:53.733750 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:53.733715 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:54.110701 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:54.110658 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:56:54.207553 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:56:54.207524 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:06.244241 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:06.244200 2526 patch_prober.go:28] interesting pod/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Aug 04 18:57:06.244652 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:06.244265 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" podUID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Aug 04 18:57:06.321110 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:06.321083 2526 generic.go:356] "Generic (PLEG): container finished" podID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" containerID="8759126f45b6ccd4db10f6c838ac4d013fcb706ca148e6fe5cc4666f02a13d73" exitCode=0 Aug 04 18:57:06.321255 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:06.321160 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" event={"ID":"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9","Type":"ContainerDied","Data":"8759126f45b6ccd4db10f6c838ac4d013fcb706ca148e6fe5cc4666f02a13d73"} Aug 04 18:57:06.321552 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:06.321535 2526 scope.go:117] "RemoveContainer" containerID="8759126f45b6ccd4db10f6c838ac4d013fcb706ca148e6fe5cc4666f02a13d73" Aug 04 18:57:06.322539 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:06.322489 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:57:06.322639 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:06.322595 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:57:06.322710 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:06.322641 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:57:07.328244 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:07.328205 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" event={"ID":"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9","Type":"ContainerStarted","Data":"5e11262bfc092fd5706bb033255dadc0ff7b6911bf1379966ea826f3a8cdb935"} Aug 04 18:57:08.272369 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:08.272334 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:57:08.272563 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:08.272468 2526 patch_prober.go:28] interesting pod/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Aug 04 18:57:08.272563 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:08.272521 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" podUID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Aug 04 18:57:09.273078 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:09.273041 2526 patch_prober.go:28] interesting pod/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Aug 04 18:57:09.273471 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:09.273094 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" podUID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Aug 04 18:57:10.272549 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:10.272512 2526 patch_prober.go:28] interesting pod/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Aug 04 18:57:10.272716 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:10.272577 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" podUID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Aug 04 18:57:11.272883 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:11.272846 2526 patch_prober.go:28] interesting pod/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Aug 04 18:57:11.273287 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:11.272905 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" podUID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Aug 04 18:57:12.272715 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:12.272680 2526 patch_prober.go:28] interesting pod/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Aug 04 18:57:12.272889 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:12.272736 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" podUID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Aug 04 18:57:13.276129 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:13.276100 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:57:13.276543 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:13.276440 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:57:13.277192 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:13.277170 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:57:17.501210 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:17.501177 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:36.280026 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:36.279993 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:36.367892 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:36.367835 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:36.466637 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:36.466603 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:36.552751 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:36.552644 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:36.916272 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:36.916170 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:37.332024 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:37.331990 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:37.404005 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:37.403958 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:43.814526 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:43.814492 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22"] Aug 04 18:57:43.815103 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:43.814719 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" podUID="c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" containerName="istio-proxy" containerID="cri-o://3aa28e4f01512a8a76f4d45e4d49d0a9f09512dd12b83985efd2d4dabac3fb3d" gracePeriod=30 Aug 04 18:57:45.641726 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:45.641683 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:45.743255 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:45.743209 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:48.381323 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:48.381263 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:49.065645 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.065621 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:57:49.197759 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.197692 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp4hj\" (UniqueName: \"kubernetes.io/projected/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-kube-api-access-fp4hj\") pod \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " Aug 04 18:57:49.197759 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.197720 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-podinfo\") pod \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " Aug 04 18:57:49.197759 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.197748 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-token\") pod \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " Aug 04 18:57:49.198064 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.197772 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-credential-socket\") pod \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " Aug 04 18:57:49.198064 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.197818 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-workload-certs\") pod \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " Aug 04 18:57:49.198064 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.197836 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-envoy\") pod \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " Aug 04 18:57:49.198064 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.197863 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istiod-ca-cert\") pod \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " Aug 04 18:57:49.198064 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.197916 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-workload-socket\") pod \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " Aug 04 18:57:49.198064 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.197942 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-data\") pod \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\" (UID: \"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4\") " Aug 04 18:57:49.198386 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.198213 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" (UID: "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:57:49.198447 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.198407 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-data" (OuterVolumeSpecName: "istio-data") pod "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" (UID: "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:57:49.198447 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.198422 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" (UID: "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:57:49.198561 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.198483 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" (UID: "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:57:49.198561 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.198549 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" (UID: "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:57:49.199998 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.199947 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" (UID: "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:57:49.200147 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.200090 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" (UID: "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:57:49.200384 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.200363 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-token" (OuterVolumeSpecName: "istio-token") pod "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" (UID: "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:57:49.200384 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.200369 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-kube-api-access-fp4hj" (OuterVolumeSpecName: "kube-api-access-fp4hj") pod "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" (UID: "c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4"). InnerVolumeSpecName "kube-api-access-fp4hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:57:49.298975 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.298949 2526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-workload-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:57:49.299091 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.298987 2526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-data\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:57:49.299091 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.299000 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp4hj\" (UniqueName: \"kubernetes.io/projected/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-kube-api-access-fp4hj\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:57:49.299091 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.299011 2526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-podinfo\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:57:49.299091 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.299019 2526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-token\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:57:49.299091 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.299027 2526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-credential-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:57:49.299091 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.299035 2526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-workload-certs\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:57:49.299091 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.299044 2526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istio-envoy\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:57:49.299091 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.299051 2526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4-istiod-ca-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:57:49.546153 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.546121 2526 generic.go:356] "Generic (PLEG): container finished" podID="c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" containerID="3aa28e4f01512a8a76f4d45e4d49d0a9f09512dd12b83985efd2d4dabac3fb3d" exitCode=0 Aug 04 18:57:49.546611 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.546165 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" event={"ID":"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4","Type":"ContainerDied","Data":"3aa28e4f01512a8a76f4d45e4d49d0a9f09512dd12b83985efd2d4dabac3fb3d"} Aug 04 18:57:49.546611 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.546190 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" Aug 04 18:57:49.546611 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.546247 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22" event={"ID":"c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4","Type":"ContainerDied","Data":"2c032ca6caa3fe2a71340b47a8d00739ae8e23c47611e051689e0e4639a06f3e"} Aug 04 18:57:49.546611 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.546268 2526 scope.go:117] "RemoveContainer" containerID="3aa28e4f01512a8a76f4d45e4d49d0a9f09512dd12b83985efd2d4dabac3fb3d" Aug 04 18:57:49.555062 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.555044 2526 scope.go:117] "RemoveContainer" containerID="0e81df0709a1c888d62fe3b7a15ae45213505ffbcec65d036ef2f514378be768" Aug 04 18:57:49.562578 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.562563 2526 scope.go:117] "RemoveContainer" containerID="3aa28e4f01512a8a76f4d45e4d49d0a9f09512dd12b83985efd2d4dabac3fb3d" Aug 04 18:57:49.562804 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:57:49.562788 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa28e4f01512a8a76f4d45e4d49d0a9f09512dd12b83985efd2d4dabac3fb3d\": container with ID starting with 3aa28e4f01512a8a76f4d45e4d49d0a9f09512dd12b83985efd2d4dabac3fb3d not found: ID does not exist" containerID="3aa28e4f01512a8a76f4d45e4d49d0a9f09512dd12b83985efd2d4dabac3fb3d" Aug 04 18:57:49.562851 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.562811 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa28e4f01512a8a76f4d45e4d49d0a9f09512dd12b83985efd2d4dabac3fb3d"} err="failed to get container status \"3aa28e4f01512a8a76f4d45e4d49d0a9f09512dd12b83985efd2d4dabac3fb3d\": rpc error: code = NotFound desc = could not find container \"3aa28e4f01512a8a76f4d45e4d49d0a9f09512dd12b83985efd2d4dabac3fb3d\": container with ID starting with 3aa28e4f01512a8a76f4d45e4d49d0a9f09512dd12b83985efd2d4dabac3fb3d not found: ID does not exist" Aug 04 18:57:49.562851 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.562826 2526 scope.go:117] "RemoveContainer" containerID="0e81df0709a1c888d62fe3b7a15ae45213505ffbcec65d036ef2f514378be768" Aug 04 18:57:49.563036 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:57:49.563019 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e81df0709a1c888d62fe3b7a15ae45213505ffbcec65d036ef2f514378be768\": container with ID starting with 0e81df0709a1c888d62fe3b7a15ae45213505ffbcec65d036ef2f514378be768 not found: ID does not exist" containerID="0e81df0709a1c888d62fe3b7a15ae45213505ffbcec65d036ef2f514378be768" Aug 04 18:57:49.563082 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.563039 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e81df0709a1c888d62fe3b7a15ae45213505ffbcec65d036ef2f514378be768"} err="failed to get container status \"0e81df0709a1c888d62fe3b7a15ae45213505ffbcec65d036ef2f514378be768\": rpc error: code = NotFound desc = could not find container \"0e81df0709a1c888d62fe3b7a15ae45213505ffbcec65d036ef2f514378be768\": container with ID starting with 0e81df0709a1c888d62fe3b7a15ae45213505ffbcec65d036ef2f514378be768 not found: ID does not exist" Aug 04 18:57:49.571163 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.571142 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22"] Aug 04 18:57:49.577386 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:49.577364 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-95504a-openshift-default-54f6f8b9d8-jtw22"] Aug 04 18:57:51.181218 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:51.181186 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" path="/var/lib/kubelet/pods/c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4/volumes" Aug 04 18:57:53.782654 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:53.782624 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:53.862601 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:53.862511 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:53.963573 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:53.963546 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:54.048663 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:54.048566 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:57:54.374616 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:57:54.374530 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:00.824987 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:00.824934 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:00.925957 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:00.925927 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:01.090227 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:01.090148 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2"] Aug 04 18:58:01.090432 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:01.090408 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" podUID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" containerName="istio-proxy" containerID="cri-o://5e11262bfc092fd5706bb033255dadc0ff7b6911bf1379966ea826f3a8cdb935" gracePeriod=30 Aug 04 18:58:01.121248 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:01.121217 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:01.345777 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:01.345675 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:01.714544 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:01.714510 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:02.022609 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.022520 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj"] Aug 04 18:58:02.023190 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.023171 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" containerName="istio-proxy" Aug 04 18:58:02.023190 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.023191 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" containerName="istio-proxy" Aug 04 18:58:02.023282 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.023200 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerName="istio-proxy" Aug 04 18:58:02.023282 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.023207 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="788ce345-6bc3-45fb-b42f-67b8361a3f90" containerName="istio-proxy" Aug 04 18:58:02.023282 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.023216 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" containerName="istio-proxy" Aug 04 18:58:02.023282 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.023224 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" containerName="istio-proxy" Aug 04 18:58:02.023404 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.023317 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" containerName="istio-proxy" Aug 04 18:58:02.025322 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.025305 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.027897 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.027878 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-219d48-openshift-default-dockercfg-d8xd9\"" Aug 04 18:58:02.044185 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.044158 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj"] Aug 04 18:58:02.052607 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.052584 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:02.112163 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.112121 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-istio-envoy\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.112388 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.112176 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-istio-data\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.112388 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.112297 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0b1d7c-d787-453d-9656-949d489b2c13-istiod-ca-cert\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.112388 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.112340 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj7t8\" (UniqueName: \"kubernetes.io/projected/de0b1d7c-d787-453d-9656-949d489b2c13-kube-api-access-tj7t8\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.112563 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.112393 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0b1d7c-d787-453d-9656-949d489b2c13-istio-podinfo\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.112563 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.112452 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-workload-certs\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.112563 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.112495 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-workload-socket\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.112563 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.112540 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0b1d7c-d787-453d-9656-949d489b2c13-istio-token\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.112774 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.112623 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-credential-socket\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.213374 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.213326 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0b1d7c-d787-453d-9656-949d489b2c13-istiod-ca-cert\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.213374 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.213379 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj7t8\" (UniqueName: \"kubernetes.io/projected/de0b1d7c-d787-453d-9656-949d489b2c13-kube-api-access-tj7t8\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.213638 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.213413 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0b1d7c-d787-453d-9656-949d489b2c13-istio-podinfo\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.213638 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.213463 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-workload-certs\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.213752 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.213632 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-workload-socket\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.213752 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.213710 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0b1d7c-d787-453d-9656-949d489b2c13-istio-token\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.213855 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.213753 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-credential-socket\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.213855 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.213810 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-istio-envoy\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.213855 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.213834 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-workload-certs\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.213855 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.213844 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-istio-data\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.214079 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.214036 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-workload-socket\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.214221 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.214197 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0b1d7c-d787-453d-9656-949d489b2c13-istiod-ca-cert\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.214358 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.214253 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-credential-socket\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.214358 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.214340 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-istio-data\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.215912 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.215890 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-istio-envoy\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.216263 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.216247 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0b1d7c-d787-453d-9656-949d489b2c13-istio-podinfo\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.222863 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.222841 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0b1d7c-d787-453d-9656-949d489b2c13-istio-token\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.223217 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.223196 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj7t8\" (UniqueName: \"kubernetes.io/projected/de0b1d7c-d787-453d-9656-949d489b2c13-kube-api-access-tj7t8\") pod \"e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.335042 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.334931 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:02.479079 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.479055 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj"] Aug 04 18:58:02.481265 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.481238 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:02.481880 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:58:02.481857 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0b1d7c_d787_453d_9656_949d489b2c13.slice/crio-40750a107f8e4321be87c108fb9283a73a1268109a2d18e2914b6048866e9d47 WatchSource:0}: Error finding container 40750a107f8e4321be87c108fb9283a73a1268109a2d18e2914b6048866e9d47: Status 404 returned error can't find the container with id 40750a107f8e4321be87c108fb9283a73a1268109a2d18e2914b6048866e9d47 Aug 04 18:58:02.483886 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.483852 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:58:02.484004 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.483958 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:58:02.484049 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.484038 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:58:02.614951 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.614863 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" event={"ID":"de0b1d7c-d787-453d-9656-949d489b2c13","Type":"ContainerStarted","Data":"5435b7d7b8a49a9d51589106a5e8f5b7013a56554e8649b26dd8b4f1d68f207b"} Aug 04 18:58:02.614951 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.614926 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" event={"ID":"de0b1d7c-d787-453d-9656-949d489b2c13","Type":"ContainerStarted","Data":"40750a107f8e4321be87c108fb9283a73a1268109a2d18e2914b6048866e9d47"} Aug 04 18:58:02.640261 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:02.640201 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" podStartSLOduration=0.640184099 podStartE2EDuration="640.184099ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:58:02.638152549 +0000 UTC m=+2839.889435094" watchObservedRunningTime="2026-08-04 18:58:02.640184099 +0000 UTC m=+2839.891466691" Aug 04 18:58:03.335452 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:03.335422 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:03.341011 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:03.340959 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:03.620303 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:03.620183 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:03.621161 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:03.621146 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:04.614342 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:04.614306 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:04.708001 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:04.707952 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:05.743410 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:05.743381 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:05.878659 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:05.878628 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:06.341603 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.341582 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:58:06.451631 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.451609 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-workload-socket\") pod \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " Aug 04 18:58:06.451809 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.451646 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-envoy\") pod \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " Aug 04 18:58:06.451809 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.451664 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dbbm\" (UniqueName: \"kubernetes.io/projected/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-kube-api-access-4dbbm\") pod \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " Aug 04 18:58:06.451809 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.451680 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-podinfo\") pod \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " Aug 04 18:58:06.451809 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.451707 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istiod-ca-cert\") pod \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " Aug 04 18:58:06.451809 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.451743 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-data\") pod \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " Aug 04 18:58:06.451809 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.451809 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-credential-socket\") pod \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " Aug 04 18:58:06.452137 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.451847 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-token\") pod \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " Aug 04 18:58:06.452137 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.451902 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-workload-certs\") pod \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\" (UID: \"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9\") " Aug 04 18:58:06.452137 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.451947 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" (UID: "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:58:06.452137 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.452104 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" (UID: "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:58:06.452348 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.452184 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-data" (OuterVolumeSpecName: "istio-data") pod "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" (UID: "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:58:06.452348 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.452226 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" (UID: "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:58:06.452348 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.452336 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" (UID: "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:58:06.452497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.452438 2526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-workload-certs\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:58:06.452497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.452459 2526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-workload-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:58:06.452497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.452472 2526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istiod-ca-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:58:06.452497 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.452485 2526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-data\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:58:06.452708 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.452497 2526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-credential-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:58:06.453945 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.453919 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" (UID: "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:58:06.454110 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.454090 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-token" (OuterVolumeSpecName: "istio-token") pod "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" (UID: "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:58:06.454162 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.454096 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-kube-api-access-4dbbm" (OuterVolumeSpecName: "kube-api-access-4dbbm") pod "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" (UID: "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9"). InnerVolumeSpecName "kube-api-access-4dbbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:58:06.454162 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.454133 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" (UID: "4985c7a2-f2c9-4f67-a773-fb3d54a13aa9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:58:06.532067 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.532040 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:06.553872 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.553850 2526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-token\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:58:06.553872 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.553867 2526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-envoy\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:58:06.553872 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.553876 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dbbm\" (UniqueName: \"kubernetes.io/projected/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-kube-api-access-4dbbm\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:58:06.554065 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.553885 2526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9-istio-podinfo\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 18:58:06.638409 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.638378 2526 generic.go:356] "Generic (PLEG): container finished" podID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" containerID="5e11262bfc092fd5706bb033255dadc0ff7b6911bf1379966ea826f3a8cdb935" exitCode=0 Aug 04 18:58:06.638548 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.638427 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" Aug 04 18:58:06.638548 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.638450 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" event={"ID":"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9","Type":"ContainerDied","Data":"5e11262bfc092fd5706bb033255dadc0ff7b6911bf1379966ea826f3a8cdb935"} Aug 04 18:58:06.638548 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.638504 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2" event={"ID":"4985c7a2-f2c9-4f67-a773-fb3d54a13aa9","Type":"ContainerDied","Data":"e0307c56213152c1ee75eafb09b36cd8425990e9c5b71cc2241b4c8df207d4cb"} Aug 04 18:58:06.638548 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.638535 2526 scope.go:117] "RemoveContainer" containerID="5e11262bfc092fd5706bb033255dadc0ff7b6911bf1379966ea826f3a8cdb935" Aug 04 18:58:06.648116 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.648096 2526 scope.go:117] "RemoveContainer" containerID="8759126f45b6ccd4db10f6c838ac4d013fcb706ca148e6fe5cc4666f02a13d73" Aug 04 18:58:06.656090 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.656069 2526 scope.go:117] "RemoveContainer" containerID="5e11262bfc092fd5706bb033255dadc0ff7b6911bf1379966ea826f3a8cdb935" Aug 04 18:58:06.656394 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:58:06.656374 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e11262bfc092fd5706bb033255dadc0ff7b6911bf1379966ea826f3a8cdb935\": container with ID starting with 5e11262bfc092fd5706bb033255dadc0ff7b6911bf1379966ea826f3a8cdb935 not found: ID does not exist" containerID="5e11262bfc092fd5706bb033255dadc0ff7b6911bf1379966ea826f3a8cdb935" Aug 04 18:58:06.656444 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.656401 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e11262bfc092fd5706bb033255dadc0ff7b6911bf1379966ea826f3a8cdb935"} err="failed to get container status \"5e11262bfc092fd5706bb033255dadc0ff7b6911bf1379966ea826f3a8cdb935\": rpc error: code = NotFound desc = could not find container \"5e11262bfc092fd5706bb033255dadc0ff7b6911bf1379966ea826f3a8cdb935\": container with ID starting with 5e11262bfc092fd5706bb033255dadc0ff7b6911bf1379966ea826f3a8cdb935 not found: ID does not exist" Aug 04 18:58:06.656444 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.656419 2526 scope.go:117] "RemoveContainer" containerID="8759126f45b6ccd4db10f6c838ac4d013fcb706ca148e6fe5cc4666f02a13d73" Aug 04 18:58:06.656674 ip-10-0-136-165 kubenswrapper[2526]: E0804 18:58:06.656657 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8759126f45b6ccd4db10f6c838ac4d013fcb706ca148e6fe5cc4666f02a13d73\": container with ID starting with 8759126f45b6ccd4db10f6c838ac4d013fcb706ca148e6fe5cc4666f02a13d73 not found: ID does not exist" containerID="8759126f45b6ccd4db10f6c838ac4d013fcb706ca148e6fe5cc4666f02a13d73" Aug 04 18:58:06.656722 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.656682 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8759126f45b6ccd4db10f6c838ac4d013fcb706ca148e6fe5cc4666f02a13d73"} err="failed to get container status \"8759126f45b6ccd4db10f6c838ac4d013fcb706ca148e6fe5cc4666f02a13d73\": rpc error: code = NotFound desc = could not find container \"8759126f45b6ccd4db10f6c838ac4d013fcb706ca148e6fe5cc4666f02a13d73\": container with ID starting with 8759126f45b6ccd4db10f6c838ac4d013fcb706ca148e6fe5cc4666f02a13d73 not found: ID does not exist" Aug 04 18:58:06.676403 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.676383 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2"] Aug 04 18:58:06.686359 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.686332 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0de7a2-openshift-default-7bdf7457c6-wg9n2"] Aug 04 18:58:06.709109 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:06.709055 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:07.182960 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:07.182880 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" path="/var/lib/kubelet/pods/4985c7a2-f2c9-4f67-a773-fb3d54a13aa9/volumes" Aug 04 18:58:18.986805 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:18.986770 2526 patch_prober.go:28] interesting pod/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": read tcp 10.133.0.2:37996->10.133.0.60:15021: read: connection reset by peer" start-of-body= Aug 04 18:58:18.987203 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:18.986840 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" podUID="de0b1d7c-d787-453d-9656-949d489b2c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": read tcp 10.133.0.2:37996->10.133.0.60:15021: read: connection reset by peer" Aug 04 18:58:19.725987 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:19.725937 2526 generic.go:356] "Generic (PLEG): container finished" podID="de0b1d7c-d787-453d-9656-949d489b2c13" containerID="5435b7d7b8a49a9d51589106a5e8f5b7013a56554e8649b26dd8b4f1d68f207b" exitCode=0 Aug 04 18:58:19.726149 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:19.726019 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" event={"ID":"de0b1d7c-d787-453d-9656-949d489b2c13","Type":"ContainerDied","Data":"5435b7d7b8a49a9d51589106a5e8f5b7013a56554e8649b26dd8b4f1d68f207b"} Aug 04 18:58:19.726360 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:19.726346 2526 scope.go:117] "RemoveContainer" containerID="5435b7d7b8a49a9d51589106a5e8f5b7013a56554e8649b26dd8b4f1d68f207b" Aug 04 18:58:19.727277 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:19.727252 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:58:19.727334 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:19.727313 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:58:19.727369 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:19.727343 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:58:20.733296 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:20.733262 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" event={"ID":"de0b1d7c-d787-453d-9656-949d489b2c13","Type":"ContainerStarted","Data":"dd16035de02ea65a985c7dfef8950cb9ec162761f7bf3ffe49a8447bec905eca"} Aug 04 18:58:21.335196 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:21.335163 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:21.335364 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:21.335340 2526 patch_prober.go:28] interesting pod/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 18:58:21.335412 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:21.335391 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" podUID="de0b1d7c-d787-453d-9656-949d489b2c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 18:58:22.335415 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:22.335383 2526 patch_prober.go:28] interesting pod/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 18:58:22.335809 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:22.335428 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" podUID="de0b1d7c-d787-453d-9656-949d489b2c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 18:58:23.335614 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:23.335577 2526 patch_prober.go:28] interesting pod/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 18:58:23.336014 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:23.335628 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" podUID="de0b1d7c-d787-453d-9656-949d489b2c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 18:58:24.336306 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:24.336272 2526 patch_prober.go:28] interesting pod/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 18:58:24.336688 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:24.336329 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" podUID="de0b1d7c-d787-453d-9656-949d489b2c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 18:58:25.336292 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:25.336258 2526 patch_prober.go:28] interesting pod/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 18:58:25.336470 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:25.336314 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" podUID="de0b1d7c-d787-453d-9656-949d489b2c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 18:58:26.339344 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:26.339311 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:26.339724 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:26.339677 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:26.340488 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:26.340465 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 18:58:34.786721 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:34.786686 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:35.490745 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.490707 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:35.499541 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.499513 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s"] Aug 04 18:58:35.499941 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.499924 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" containerName="istio-proxy" Aug 04 18:58:35.500052 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.499943 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" containerName="istio-proxy" Aug 04 18:58:35.500110 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.500099 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1d780e6-9605-4e8c-aeb2-984a2f2d9bb4" containerName="istio-proxy" Aug 04 18:58:35.500161 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.500115 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" containerName="istio-proxy" Aug 04 18:58:35.500161 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.500126 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" containerName="istio-proxy" Aug 04 18:58:35.503096 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.503075 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.505838 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.505820 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-de2394-openshift-default-dockercfg-4slbb\"" Aug 04 18:58:35.517142 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.517116 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s"] Aug 04 18:58:35.567832 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.567797 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:35.597238 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.597200 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-workload-certs\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.597431 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.597254 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-workload-socket\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.597431 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.597294 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-credential-socket\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.597431 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.597320 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-data\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.597431 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.597356 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd77988c-6b5c-4a10-af94-9da49d0446ca-istiod-ca-cert\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.597431 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.597428 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-envoy\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.597706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.597473 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-token\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.597706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.597540 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b65lv\" (UniqueName: \"kubernetes.io/projected/fd77988c-6b5c-4a10-af94-9da49d0446ca-kube-api-access-b65lv\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.597706 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.597580 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-podinfo\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.698353 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.698322 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-token\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.698552 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.698372 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b65lv\" (UniqueName: \"kubernetes.io/projected/fd77988c-6b5c-4a10-af94-9da49d0446ca-kube-api-access-b65lv\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.698552 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.698403 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-podinfo\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.698552 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.698420 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-workload-certs\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.698552 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.698438 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-workload-socket\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.698552 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.698460 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-credential-socket\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.698552 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.698477 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-data\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.698552 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.698508 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd77988c-6b5c-4a10-af94-9da49d0446ca-istiod-ca-cert\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.698922 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.698587 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-envoy\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.698922 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.698882 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-workload-socket\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.699070 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.698921 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-workload-certs\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.699192 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.699164 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-credential-socket\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.699309 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.699284 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd77988c-6b5c-4a10-af94-9da49d0446ca-istiod-ca-cert\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.699426 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.699285 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-data\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.701142 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.701120 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-podinfo\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.701285 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.701268 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-envoy\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.707271 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.707249 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-token\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.708295 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.708250 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b65lv\" (UniqueName: \"kubernetes.io/projected/fd77988c-6b5c-4a10-af94-9da49d0446ca-kube-api-access-b65lv\") pod \"e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.763100 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.763029 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:35.813312 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.813278 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:35.979092 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.979063 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s"] Aug 04 18:58:35.980831 ip-10-0-136-165 kubenswrapper[2526]: W0804 18:58:35.980795 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd77988c_6b5c_4a10_af94_9da49d0446ca.slice/crio-ceb13ab9d1a76f0737fd4cadeaf0cd10efc7587a684c7af36868d543363c5d6a WatchSource:0}: Error finding container ceb13ab9d1a76f0737fd4cadeaf0cd10efc7587a684c7af36868d543363c5d6a: Status 404 returned error can't find the container with id ceb13ab9d1a76f0737fd4cadeaf0cd10efc7587a684c7af36868d543363c5d6a Aug 04 18:58:35.982991 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.982946 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:58:35.983051 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.983025 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:58:35.983092 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:35.983055 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:58:36.829081 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:36.829043 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" event={"ID":"fd77988c-6b5c-4a10-af94-9da49d0446ca","Type":"ContainerStarted","Data":"91a24d907dc6a988efa2f2d0006e4d955f775a5aa4d879f8e48e8d3303cc9d82"} Aug 04 18:58:36.829460 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:36.829100 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" event={"ID":"fd77988c-6b5c-4a10-af94-9da49d0446ca","Type":"ContainerStarted","Data":"ceb13ab9d1a76f0737fd4cadeaf0cd10efc7587a684c7af36868d543363c5d6a"} Aug 04 18:58:36.856077 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:36.856030 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" podStartSLOduration=1.856014929 podStartE2EDuration="1.856014929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:58:36.854390651 +0000 UTC m=+2874.105673198" watchObservedRunningTime="2026-08-04 18:58:36.856014929 +0000 UTC m=+2874.107297474" Aug 04 18:58:37.814764 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:37.814730 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:37.819580 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:37.819557 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:37.833554 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:37.833535 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:37.834740 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:37.834726 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:39.052544 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:39.052510 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:39.170625 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:39.170597 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:40.061385 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:40.061353 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:40.316084 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:40.316002 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:40.569694 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:40.569619 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:40.663359 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:40.663331 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:58:52.839759 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:52.839725 2526 patch_prober.go:28] interesting pod/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": read tcp 10.133.0.2:52968->10.133.0.61:15021: read: connection reset by peer" start-of-body= Aug 04 18:58:52.840188 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:52.839802 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" podUID="fd77988c-6b5c-4a10-af94-9da49d0446ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": read tcp 10.133.0.2:52968->10.133.0.61:15021: read: connection reset by peer" Aug 04 18:58:52.917932 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:52.917903 2526 generic.go:356] "Generic (PLEG): container finished" podID="fd77988c-6b5c-4a10-af94-9da49d0446ca" containerID="91a24d907dc6a988efa2f2d0006e4d955f775a5aa4d879f8e48e8d3303cc9d82" exitCode=0 Aug 04 18:58:52.918083 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:52.917996 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" event={"ID":"fd77988c-6b5c-4a10-af94-9da49d0446ca","Type":"ContainerDied","Data":"91a24d907dc6a988efa2f2d0006e4d955f775a5aa4d879f8e48e8d3303cc9d82"} Aug 04 18:58:52.918454 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:52.918437 2526 scope.go:117] "RemoveContainer" containerID="91a24d907dc6a988efa2f2d0006e4d955f775a5aa4d879f8e48e8d3303cc9d82" Aug 04 18:58:52.919512 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:52.919483 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:58:52.919568 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:52.919550 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:58:52.919608 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:52.919585 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 18:58:53.924894 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:53.924858 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" event={"ID":"fd77988c-6b5c-4a10-af94-9da49d0446ca","Type":"ContainerStarted","Data":"e80c1a3c8a29e377cb39531b4db4863f6cb791994c8dc367f92c2ed878767d22"} Aug 04 18:58:54.814770 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:54.814732 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:54.815009 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:54.814933 2526 patch_prober.go:28] interesting pod/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 04 18:58:54.815103 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:54.815026 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" podUID="fd77988c-6b5c-4a10-af94-9da49d0446ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 04 18:58:55.814607 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:55.814570 2526 patch_prober.go:28] interesting pod/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 04 18:58:55.815037 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:55.814634 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" podUID="fd77988c-6b5c-4a10-af94-9da49d0446ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 04 18:58:56.815261 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:56.815223 2526 patch_prober.go:28] interesting pod/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 04 18:58:56.815673 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:56.815286 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" podUID="fd77988c-6b5c-4a10-af94-9da49d0446ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 04 18:58:57.814799 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:57.814764 2526 patch_prober.go:28] interesting pod/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 04 18:58:57.815014 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:57.814836 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" podUID="fd77988c-6b5c-4a10-af94-9da49d0446ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 04 18:58:58.845343 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:58.845308 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:58.845826 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:58.845803 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:58:58.846610 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:58:58.846591 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 18:59:03.889839 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:59:03.889791 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:59:07.657692 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:59:07.657657 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 18:59:39.557358 ip-10-0-136-165 kubenswrapper[2526]: I0804 18:59:39.557325 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:00:44.484786 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:00:44.484751 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:00:44.552598 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:00:44.552562 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:01:39.716152 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:01:39.716062 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:01:41.376294 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:01:41.376245 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:03:05.243891 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:03:05.243808 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:03:05.334096 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:03:05.334066 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:03:30.817159 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:03:30.817120 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:03:31.663407 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:03:31.663371 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:03:54.638431 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:03:54.638396 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:04:11.552030 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:04:11.551993 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:05:18.562378 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:05:18.562343 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:05:19.455162 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:05:19.455133 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:05:32.149898 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:05:32.149864 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:05:35.887894 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:05:35.887858 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:06:20.417558 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:06:20.417442 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:07:20.938869 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:20.938835 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:07:27.844890 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:27.844844 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:07:28.641060 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:28.641029 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:07:48.539648 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.539605 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:07:48.833306 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.833233 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc"] Aug 04 19:07:48.833618 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.833606 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" containerName="istio-proxy" Aug 04 19:07:48.833661 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.833619 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="4985c7a2-f2c9-4f67-a773-fb3d54a13aa9" containerName="istio-proxy" Aug 04 19:07:48.836932 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.836911 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:48.840515 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.840494 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0c0dc7-openshift-default-dockercfg-jqxsr\"" Aug 04 19:07:48.852224 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.852198 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc"] Aug 04 19:07:48.859154 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.859124 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:07:48.941047 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.940667 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-istio-data\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:48.941047 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.940713 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-credential-socket\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:48.941047 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.940825 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-workload-certs\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:48.941047 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.940856 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-workload-socket\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:48.941047 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.940923 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf5jw\" (UniqueName: \"kubernetes.io/projected/c663496f-8313-4a90-854d-b5dd2bed999d-kube-api-access-bf5jw\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:48.941508 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.941059 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c663496f-8313-4a90-854d-b5dd2bed999d-istiod-ca-cert\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:48.941508 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.941091 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c663496f-8313-4a90-854d-b5dd2bed999d-istio-token\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:48.941508 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.941129 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c663496f-8313-4a90-854d-b5dd2bed999d-istio-podinfo\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:48.941508 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.941271 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-istio-envoy\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:48.994734 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:48.994693 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:07:49.042395 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.042360 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c663496f-8313-4a90-854d-b5dd2bed999d-istiod-ca-cert\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.042395 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.042395 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c663496f-8313-4a90-854d-b5dd2bed999d-istio-token\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.042664 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.042419 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c663496f-8313-4a90-854d-b5dd2bed999d-istio-podinfo\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.042664 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.042468 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-istio-envoy\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.042664 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.042495 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-istio-data\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.042664 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.042518 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-credential-socket\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.042664 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.042555 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-workload-certs\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.042664 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.042585 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-workload-socket\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.042664 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.042618 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf5jw\" (UniqueName: \"kubernetes.io/projected/c663496f-8313-4a90-854d-b5dd2bed999d-kube-api-access-bf5jw\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.043057 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.043031 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-istio-data\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.043123 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.043066 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-workload-certs\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.043123 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.043042 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-credential-socket\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.043251 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.043229 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-workload-socket\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.043311 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.043277 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c663496f-8313-4a90-854d-b5dd2bed999d-istiod-ca-cert\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.044855 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.044836 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-istio-envoy\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.045125 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.045102 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c663496f-8313-4a90-854d-b5dd2bed999d-istio-podinfo\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.052391 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.052371 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c663496f-8313-4a90-854d-b5dd2bed999d-istio-token\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.052763 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.052731 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf5jw\" (UniqueName: \"kubernetes.io/projected/c663496f-8313-4a90-854d-b5dd2bed999d-kube-api-access-bf5jw\") pod \"e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.147173 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.147114 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:49.292850 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.292826 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc"] Aug 04 19:07:49.294513 ip-10-0-136-165 kubenswrapper[2526]: W0804 19:07:49.294474 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc663496f_8313_4a90_854d_b5dd2bed999d.slice/crio-f4d4d1c4ac587020360b989a156385a9f4b848f5c56409e287cacfe5de490f82 WatchSource:0}: Error finding container f4d4d1c4ac587020360b989a156385a9f4b848f5c56409e287cacfe5de490f82: Status 404 returned error can't find the container with id f4d4d1c4ac587020360b989a156385a9f4b848f5c56409e287cacfe5de490f82 Aug 04 19:07:49.296985 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.296924 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 19:07:49.297079 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.297059 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 19:07:49.297121 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.297105 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 19:07:49.620478 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.620443 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" event={"ID":"c663496f-8313-4a90-854d-b5dd2bed999d","Type":"ContainerStarted","Data":"58d69a9a8c19aa296d8642d5e304e0b1b58fed50ed89160094fe02e7cc213f5b"} Aug 04 19:07:49.620852 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.620522 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" event={"ID":"c663496f-8313-4a90-854d-b5dd2bed999d","Type":"ContainerStarted","Data":"f4d4d1c4ac587020360b989a156385a9f4b848f5c56409e287cacfe5de490f82"} Aug 04 19:07:49.656679 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:49.656633 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" podStartSLOduration=1.656617829 podStartE2EDuration="1.656617829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:07:49.656437712 +0000 UTC m=+3426.907720297" watchObservedRunningTime="2026-08-04 19:07:49.656617829 +0000 UTC m=+3426.907900375" Aug 04 19:07:50.147780 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:50.147745 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:50.153506 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:50.153482 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:50.625269 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:50.625236 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:50.626191 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:50.626171 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:07:51.686659 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:51.686630 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:07:51.790327 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:51.790293 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:07:52.904614 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:52.904585 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:07:53.574022 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:53.573991 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:07:53.674934 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:53.674905 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:07:53.970329 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:07:53.970296 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:08:06.472641 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:06.472594 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:08:06.667105 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:06.667071 2526 patch_prober.go:28] interesting pod/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Aug 04 19:08:06.667274 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:06.667137 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" podUID="c663496f-8313-4a90-854d-b5dd2bed999d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 04 19:08:07.222352 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:07.222322 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:08:07.591014 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:07.590904 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:08:07.719952 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:07.719909 2526 generic.go:356] "Generic (PLEG): container finished" podID="c663496f-8313-4a90-854d-b5dd2bed999d" containerID="58d69a9a8c19aa296d8642d5e304e0b1b58fed50ed89160094fe02e7cc213f5b" exitCode=0 Aug 04 19:08:07.720158 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:07.720057 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" event={"ID":"c663496f-8313-4a90-854d-b5dd2bed999d","Type":"ContainerDied","Data":"58d69a9a8c19aa296d8642d5e304e0b1b58fed50ed89160094fe02e7cc213f5b"} Aug 04 19:08:07.720602 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:07.720574 2526 scope.go:117] "RemoveContainer" containerID="58d69a9a8c19aa296d8642d5e304e0b1b58fed50ed89160094fe02e7cc213f5b" Aug 04 19:08:07.721829 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:07.721794 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 19:08:07.721937 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:07.721865 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 19:08:07.721937 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:07.721899 2526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 19:08:08.520880 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:08.520845 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:08:08.727097 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:08.727063 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" event={"ID":"c663496f-8313-4a90-854d-b5dd2bed999d","Type":"ContainerStarted","Data":"d45bd7cd672de2f9ca16d333d71d47c097f7b84bf6e90ba9a0360412ab417af7"} Aug 04 19:08:09.148263 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:09.148217 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:08:09.148475 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:09.148380 2526 patch_prober.go:28] interesting pod/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 04 19:08:09.148475 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:09.148427 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" podUID="c663496f-8313-4a90-854d-b5dd2bed999d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 04 19:08:10.148375 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:10.148340 2526 patch_prober.go:28] interesting pod/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 04 19:08:10.148942 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:10.148911 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" podUID="c663496f-8313-4a90-854d-b5dd2bed999d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 04 19:08:11.147821 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:11.147778 2526 patch_prober.go:28] interesting pod/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 04 19:08:11.148111 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:11.147848 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" podUID="c663496f-8313-4a90-854d-b5dd2bed999d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 04 19:08:12.147985 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:12.147923 2526 patch_prober.go:28] interesting pod/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 04 19:08:12.148476 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:12.148011 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" podUID="c663496f-8313-4a90-854d-b5dd2bed999d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 04 19:08:13.148306 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:13.148272 2526 patch_prober.go:28] interesting pod/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 04 19:08:13.148893 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:13.148862 2526 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" podUID="c663496f-8313-4a90-854d-b5dd2bed999d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 04 19:08:13.945302 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:13.945266 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:08:14.098593 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:14.098561 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:08:14.150929 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:14.150900 2526 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:08:14.151433 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:14.151322 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:08:14.152058 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:14.152042 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:08:14.181276 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:14.181243 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:08:16.593448 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.593413 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f"] Aug 04 19:08:16.596776 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.596752 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" Aug 04 19:08:16.599945 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.599922 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4b6c58-serving-cert\"" Aug 04 19:08:16.615476 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.615450 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f"] Aug 04 19:08:16.666658 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.666631 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:08:16.706218 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.706180 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f\" (UID: \"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" Aug 04 19:08:16.706413 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.706226 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwkf9\" (UniqueName: \"kubernetes.io/projected/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-kube-api-access-dwkf9\") pod \"maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f\" (UID: \"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" Aug 04 19:08:16.706413 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.706267 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f\" (UID: \"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" Aug 04 19:08:16.807301 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.807272 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f\" (UID: \"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" Aug 04 19:08:16.807493 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.807310 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwkf9\" (UniqueName: \"kubernetes.io/projected/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-kube-api-access-dwkf9\") pod \"maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f\" (UID: \"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" Aug 04 19:08:16.807493 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.807359 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f\" (UID: \"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" Aug 04 19:08:16.807890 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.807866 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f\" (UID: \"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" Aug 04 19:08:16.809712 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.809692 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f\" (UID: \"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" Aug 04 19:08:16.822008 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.821986 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwkf9\" (UniqueName: \"kubernetes.io/projected/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-kube-api-access-dwkf9\") pod \"maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f\" (UID: \"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" Aug 04 19:08:16.840467 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.840445 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:08:16.908478 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:16.908419 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" Aug 04 19:08:17.051990 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.051955 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f"] Aug 04 19:08:17.053659 ip-10-0-136-165 kubenswrapper[2526]: W0804 19:08:17.053633 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1868f7_bb42_4cf1_8f43_0d7cc5ff302c.slice/crio-47c9a7ed0d9b5fd799f503af9b0118c35a123282b13a63e5316c9f1b42c41ca9 WatchSource:0}: Error finding container 47c9a7ed0d9b5fd799f503af9b0118c35a123282b13a63e5316c9f1b42c41ca9: Status 404 returned error can't find the container with id 47c9a7ed0d9b5fd799f503af9b0118c35a123282b13a63e5316c9f1b42c41ca9 Aug 04 19:08:17.054872 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.054855 2526 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 19:08:17.502090 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.502053 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:08:17.602466 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.602436 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:08:17.664854 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.664784 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk"] Aug 04 19:08:17.668519 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.668499 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" Aug 04 19:08:17.671283 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.671259 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4b6c58\"" Aug 04 19:08:17.671415 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.671400 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4b6c58-dockercfg-nz5sl\"" Aug 04 19:08:17.681847 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.681826 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk"] Aug 04 19:08:17.779403 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.779367 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" event={"ID":"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c","Type":"ContainerStarted","Data":"0551d84ebb8d3f202b112dd34711b7cc4d2b030b73d3b2f9a3d050c213afbae8"} Aug 04 19:08:17.779586 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.779424 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" event={"ID":"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c","Type":"ContainerStarted","Data":"47c9a7ed0d9b5fd799f503af9b0118c35a123282b13a63e5316c9f1b42c41ca9"} Aug 04 19:08:17.779586 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.779533 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" Aug 04 19:08:17.799798 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.799743 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" podStartSLOduration=1.4535733450000001 podStartE2EDuration="1.799723968s" totalImagesPullingTime="346.150623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:08:17.79848408 +0000 UTC m=+3455.049766628" watchObservedRunningTime="2026-08-04 19:08:17.799723968 +0000 UTC m=+3455.051006520" Aug 04 19:08:17.818068 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.818040 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3290241b-e9b7-4fce-b3eb-bc729b3cf49b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk\" (UID: \"3290241b-e9b7-4fce-b3eb-bc729b3cf49b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" Aug 04 19:08:17.818203 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.818097 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdglz\" (UniqueName: \"kubernetes.io/projected/3290241b-e9b7-4fce-b3eb-bc729b3cf49b-kube-api-access-vdglz\") pod \"payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk\" (UID: \"3290241b-e9b7-4fce-b3eb-bc729b3cf49b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" Aug 04 19:08:17.919180 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.919092 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3290241b-e9b7-4fce-b3eb-bc729b3cf49b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk\" (UID: \"3290241b-e9b7-4fce-b3eb-bc729b3cf49b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" Aug 04 19:08:17.919358 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.919334 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdglz\" (UniqueName: \"kubernetes.io/projected/3290241b-e9b7-4fce-b3eb-bc729b3cf49b-kube-api-access-vdglz\") pod \"payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk\" (UID: \"3290241b-e9b7-4fce-b3eb-bc729b3cf49b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" Aug 04 19:08:17.919811 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.919794 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3290241b-e9b7-4fce-b3eb-bc729b3cf49b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk\" (UID: \"3290241b-e9b7-4fce-b3eb-bc729b3cf49b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" Aug 04 19:08:17.929831 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.929805 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdglz\" (UniqueName: \"kubernetes.io/projected/3290241b-e9b7-4fce-b3eb-bc729b3cf49b-kube-api-access-vdglz\") pod \"payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk\" (UID: \"3290241b-e9b7-4fce-b3eb-bc729b3cf49b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" Aug 04 19:08:17.979210 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:17.979189 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" Aug 04 19:08:18.124901 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:18.124875 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk"] Aug 04 19:08:18.126546 ip-10-0-136-165 kubenswrapper[2526]: W0804 19:08:18.126509 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3290241b_e9b7_4fce_b3eb_bc729b3cf49b.slice/crio-d6ef908c3394ba301c573f8052198e258916ac225788450c276dfcb5aa111afe WatchSource:0}: Error finding container d6ef908c3394ba301c573f8052198e258916ac225788450c276dfcb5aa111afe: Status 404 returned error can't find the container with id d6ef908c3394ba301c573f8052198e258916ac225788450c276dfcb5aa111afe Aug 04 19:08:18.786459 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:18.786421 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" event={"ID":"3290241b-e9b7-4fce-b3eb-bc729b3cf49b","Type":"ContainerStarted","Data":"747815af96ce072baa7fbd5998ec5cabd1f34722e5353440ab3c3d2d72d5a451"} Aug 04 19:08:18.786874 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:18.786483 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" event={"ID":"3290241b-e9b7-4fce-b3eb-bc729b3cf49b","Type":"ContainerStarted","Data":"d6ef908c3394ba301c573f8052198e258916ac225788450c276dfcb5aa111afe"} Aug 04 19:08:18.786874 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:18.786715 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" Aug 04 19:08:18.833481 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:18.833430 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" podStartSLOduration=1.833416537 podStartE2EDuration="1.833416537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:08:18.830681172 +0000 UTC m=+3456.081963730" watchObservedRunningTime="2026-08-04 19:08:18.833416537 +0000 UTC m=+3456.084699084" Aug 04 19:08:23.791427 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:23.791396 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" Aug 04 19:08:28.826846 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:28.826774 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:08:29.796881 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:29.796853 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" Aug 04 19:08:42.079297 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:42.079265 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-c9kjx_3346ba4e-c7fa-4d34-83f7-2780f3b42d33/payload-processing/0.log" Aug 04 19:08:44.554551 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:08:44.554516 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:20.193007 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:20.192914 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-c9kjx_3346ba4e-c7fa-4d34-83f7-2780f3b42d33/payload-processing/0.log" Aug 04 19:09:20.655586 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:20.655548 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:21.084144 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:21.084110 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:21.484413 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:21.484383 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:24.532425 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:24.532386 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:24.785026 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:24.784943 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:27.848645 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:27.848611 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb"] Aug 04 19:09:27.851543 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:27.851522 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:27.855165 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:27.855148 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-c75b82-serving-cert\"" Aug 04 19:09:27.866828 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:27.866807 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb"] Aug 04 19:09:27.920422 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:27.920394 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:27.969376 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:27.969344 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/445d2a74-f641-4fd9-8c11-6808510ef1e2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb\" (UID: \"445d2a74-f641-4fd9-8c11-6808510ef1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:27.969572 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:27.969404 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88z6c\" (UniqueName: \"kubernetes.io/projected/445d2a74-f641-4fd9-8c11-6808510ef1e2-kube-api-access-88z6c\") pod \"maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb\" (UID: \"445d2a74-f641-4fd9-8c11-6808510ef1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:27.969572 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:27.969466 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/445d2a74-f641-4fd9-8c11-6808510ef1e2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb\" (UID: \"445d2a74-f641-4fd9-8c11-6808510ef1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:28.070728 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:28.070702 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/445d2a74-f641-4fd9-8c11-6808510ef1e2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb\" (UID: \"445d2a74-f641-4fd9-8c11-6808510ef1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:28.070890 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:28.070745 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88z6c\" (UniqueName: \"kubernetes.io/projected/445d2a74-f641-4fd9-8c11-6808510ef1e2-kube-api-access-88z6c\") pod \"maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb\" (UID: \"445d2a74-f641-4fd9-8c11-6808510ef1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:28.070890 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:28.070789 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/445d2a74-f641-4fd9-8c11-6808510ef1e2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb\" (UID: \"445d2a74-f641-4fd9-8c11-6808510ef1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:28.070890 ip-10-0-136-165 kubenswrapper[2526]: E0804 19:09:28.070860 2526 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-serving-cert: secret "maas-api-e2e-ipp-cleanup-c75b82-serving-cert" not found Aug 04 19:09:28.071098 ip-10-0-136-165 kubenswrapper[2526]: E0804 19:09:28.070926 2526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/445d2a74-f641-4fd9-8c11-6808510ef1e2-maas-api-tls podName:445d2a74-f641-4fd9-8c11-6808510ef1e2 nodeName:}" failed. No retries permitted until 2026-08-04 19:09:28.570905687 +0000 UTC m=+3525.822188228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/445d2a74-f641-4fd9-8c11-6808510ef1e2-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" (UID: "445d2a74-f641-4fd9-8c11-6808510ef1e2") : secret "maas-api-e2e-ipp-cleanup-c75b82-serving-cert" not found Aug 04 19:09:28.071501 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:28.071483 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/445d2a74-f641-4fd9-8c11-6808510ef1e2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb\" (UID: \"445d2a74-f641-4fd9-8c11-6808510ef1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:28.088143 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:28.088115 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88z6c\" (UniqueName: \"kubernetes.io/projected/445d2a74-f641-4fd9-8c11-6808510ef1e2-kube-api-access-88z6c\") pod \"maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb\" (UID: \"445d2a74-f641-4fd9-8c11-6808510ef1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:28.167059 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:28.166986 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:28.569285 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:28.569248 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:28.577040 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:28.577017 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/445d2a74-f641-4fd9-8c11-6808510ef1e2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb\" (UID: \"445d2a74-f641-4fd9-8c11-6808510ef1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:28.579465 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:28.579439 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/445d2a74-f641-4fd9-8c11-6808510ef1e2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb\" (UID: \"445d2a74-f641-4fd9-8c11-6808510ef1e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:28.718354 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:28.718325 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:28.764015 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:28.763988 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:28.932671 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:28.932647 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb"] Aug 04 19:09:28.934086 ip-10-0-136-165 kubenswrapper[2526]: W0804 19:09:28.934050 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445d2a74_f641_4fd9_8c11_6808510ef1e2.slice/crio-8ef490a4c950c263401bd38dc9930b35f651ae85cb3656ed4104880dd6f5023e WatchSource:0}: Error finding container 8ef490a4c950c263401bd38dc9930b35f651ae85cb3656ed4104880dd6f5023e: Status 404 returned error can't find the container with id 8ef490a4c950c263401bd38dc9930b35f651ae85cb3656ed4104880dd6f5023e Aug 04 19:09:29.155425 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:29.155349 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" event={"ID":"445d2a74-f641-4fd9-8c11-6808510ef1e2","Type":"ContainerStarted","Data":"8ef490a4c950c263401bd38dc9930b35f651ae85cb3656ed4104880dd6f5023e"} Aug 04 19:09:30.161831 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:30.161792 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" event={"ID":"445d2a74-f641-4fd9-8c11-6808510ef1e2","Type":"ContainerStarted","Data":"38ef908a6871815e50de77181781d0718c3c152a0c5feda414738be3a411c068"} Aug 04 19:09:30.162290 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:30.161898 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:30.192813 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:30.192758 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" podStartSLOduration=2.451524379 podStartE2EDuration="3.192742843s" totalImagesPullingTime="741.218464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:09:30.191450678 +0000 UTC m=+3527.442733223" watchObservedRunningTime="2026-08-04 19:09:30.192742843 +0000 UTC m=+3527.444025439" Aug 04 19:09:33.510525 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:33.510497 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:33.610542 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:33.610511 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:36.172171 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:36.172138 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:44.431656 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:44.431626 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:56.182140 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.182111 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb"] Aug 04 19:09:56.182648 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.182298 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" podUID="445d2a74-f641-4fd9-8c11-6808510ef1e2" containerName="maas-api" containerID="cri-o://38ef908a6871815e50de77181781d0718c3c152a0c5feda414738be3a411c068" gracePeriod=30 Aug 04 19:09:56.213265 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.213238 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:56.301826 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.301777 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:56.401908 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.401877 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:56.456321 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.456278 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:56.485314 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.485269 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:56.539675 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.539641 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/445d2a74-f641-4fd9-8c11-6808510ef1e2-maas-api-tls\") pod \"445d2a74-f641-4fd9-8c11-6808510ef1e2\" (UID: \"445d2a74-f641-4fd9-8c11-6808510ef1e2\") " Aug 04 19:09:56.539855 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.539756 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/445d2a74-f641-4fd9-8c11-6808510ef1e2-openshift-service-ca\") pod \"445d2a74-f641-4fd9-8c11-6808510ef1e2\" (UID: \"445d2a74-f641-4fd9-8c11-6808510ef1e2\") " Aug 04 19:09:56.539855 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.539823 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88z6c\" (UniqueName: \"kubernetes.io/projected/445d2a74-f641-4fd9-8c11-6808510ef1e2-kube-api-access-88z6c\") pod \"445d2a74-f641-4fd9-8c11-6808510ef1e2\" (UID: \"445d2a74-f641-4fd9-8c11-6808510ef1e2\") " Aug 04 19:09:56.540121 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.540083 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/445d2a74-f641-4fd9-8c11-6808510ef1e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "445d2a74-f641-4fd9-8c11-6808510ef1e2" (UID: "445d2a74-f641-4fd9-8c11-6808510ef1e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:09:56.541804 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.541782 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445d2a74-f641-4fd9-8c11-6808510ef1e2-kube-api-access-88z6c" (OuterVolumeSpecName: "kube-api-access-88z6c") pod "445d2a74-f641-4fd9-8c11-6808510ef1e2" (UID: "445d2a74-f641-4fd9-8c11-6808510ef1e2"). InnerVolumeSpecName "kube-api-access-88z6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:09:56.541885 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.541798 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/445d2a74-f641-4fd9-8c11-6808510ef1e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "445d2a74-f641-4fd9-8c11-6808510ef1e2" (UID: "445d2a74-f641-4fd9-8c11-6808510ef1e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:09:56.641399 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.641368 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88z6c\" (UniqueName: \"kubernetes.io/projected/445d2a74-f641-4fd9-8c11-6808510ef1e2-kube-api-access-88z6c\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:09:56.641399 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.641394 2526 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/445d2a74-f641-4fd9-8c11-6808510ef1e2-maas-api-tls\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:09:56.641399 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.641403 2526 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/445d2a74-f641-4fd9-8c11-6808510ef1e2-openshift-service-ca\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:09:56.867754 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:56.867680 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:09:57.300418 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:57.300384 2526 generic.go:356] "Generic (PLEG): container finished" podID="445d2a74-f641-4fd9-8c11-6808510ef1e2" containerID="38ef908a6871815e50de77181781d0718c3c152a0c5feda414738be3a411c068" exitCode=0 Aug 04 19:09:57.300790 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:57.300427 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" Aug 04 19:09:57.300790 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:57.300460 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" event={"ID":"445d2a74-f641-4fd9-8c11-6808510ef1e2","Type":"ContainerDied","Data":"38ef908a6871815e50de77181781d0718c3c152a0c5feda414738be3a411c068"} Aug 04 19:09:57.300790 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:57.300511 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb" event={"ID":"445d2a74-f641-4fd9-8c11-6808510ef1e2","Type":"ContainerDied","Data":"8ef490a4c950c263401bd38dc9930b35f651ae85cb3656ed4104880dd6f5023e"} Aug 04 19:09:57.300790 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:57.300532 2526 scope.go:117] "RemoveContainer" containerID="38ef908a6871815e50de77181781d0718c3c152a0c5feda414738be3a411c068" Aug 04 19:09:57.312641 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:57.312620 2526 scope.go:117] "RemoveContainer" containerID="38ef908a6871815e50de77181781d0718c3c152a0c5feda414738be3a411c068" Aug 04 19:09:57.312915 ip-10-0-136-165 kubenswrapper[2526]: E0804 19:09:57.312895 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ef908a6871815e50de77181781d0718c3c152a0c5feda414738be3a411c068\": container with ID starting with 38ef908a6871815e50de77181781d0718c3c152a0c5feda414738be3a411c068 not found: ID does not exist" containerID="38ef908a6871815e50de77181781d0718c3c152a0c5feda414738be3a411c068" Aug 04 19:09:57.312976 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:57.312923 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ef908a6871815e50de77181781d0718c3c152a0c5feda414738be3a411c068"} err="failed to get container status \"38ef908a6871815e50de77181781d0718c3c152a0c5feda414738be3a411c068\": rpc error: code = NotFound desc = could not find container \"38ef908a6871815e50de77181781d0718c3c152a0c5feda414738be3a411c068\": container with ID starting with 38ef908a6871815e50de77181781d0718c3c152a0c5feda414738be3a411c068 not found: ID does not exist" Aug 04 19:09:57.324903 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:57.324881 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb"] Aug 04 19:09:57.330170 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:57.330151 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c75b82-56fb9cc465-29xzb"] Aug 04 19:09:59.183310 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:09:59.183273 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="445d2a74-f641-4fd9-8c11-6808510ef1e2" path="/var/lib/kubelet/pods/445d2a74-f641-4fd9-8c11-6808510ef1e2/volumes" Aug 04 19:10:04.645278 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:10:04.645235 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:10:04.756096 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:10:04.756058 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:10:05.001585 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:10:05.001551 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:10:05.396564 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:10:05.396482 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:10:28.602426 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:10:28.602383 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:10:28.718907 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:10:28.718872 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:10:28.807430 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:10:28.807400 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:10:28.917581 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:10:28.917503 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:10:29.427692 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:10:29.427647 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:10:34.496516 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:10:34.496483 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:10:37.251922 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:10:37.251850 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:10:37.356550 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:10:37.356514 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:10.880601 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:10.880562 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:12.274631 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:12.274584 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:19.691597 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:19.691562 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:26.303828 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:26.303794 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:26.404706 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:26.404668 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:26.567132 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:26.567015 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc"] Aug 04 19:11:26.567636 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:26.567602 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" podUID="c663496f-8313-4a90-854d-b5dd2bed999d" containerName="istio-proxy" containerID="cri-o://d45bd7cd672de2f9ca16d333d71d47c097f7b84bf6e90ba9a0360412ab417af7" gracePeriod=30 Aug 04 19:11:26.587401 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:26.587353 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:26.851414 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:26.851327 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:29.154280 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:29.154238 2526 patch_prober.go:28] interesting pod/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 19:11:29.154680 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:29.154327 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" podUID="c663496f-8313-4a90-854d-b5dd2bed999d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 19:11:31.792544 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.792505 2526 generic.go:356] "Generic (PLEG): container finished" podID="c663496f-8313-4a90-854d-b5dd2bed999d" containerID="d45bd7cd672de2f9ca16d333d71d47c097f7b84bf6e90ba9a0360412ab417af7" exitCode=0 Aug 04 19:11:31.792934 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.792574 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" event={"ID":"c663496f-8313-4a90-854d-b5dd2bed999d","Type":"ContainerDied","Data":"d45bd7cd672de2f9ca16d333d71d47c097f7b84bf6e90ba9a0360412ab417af7"} Aug 04 19:11:31.792934 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.792653 2526 scope.go:117] "RemoveContainer" containerID="58d69a9a8c19aa296d8642d5e304e0b1b58fed50ed89160094fe02e7cc213f5b" Aug 04 19:11:31.829754 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.829732 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:11:31.951419 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.951391 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-workload-certs\") pod \"c663496f-8313-4a90-854d-b5dd2bed999d\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " Aug 04 19:11:31.951578 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.951423 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-workload-socket\") pod \"c663496f-8313-4a90-854d-b5dd2bed999d\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " Aug 04 19:11:31.951578 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.951454 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-istio-envoy\") pod \"c663496f-8313-4a90-854d-b5dd2bed999d\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " Aug 04 19:11:31.951578 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.951509 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-istio-data\") pod \"c663496f-8313-4a90-854d-b5dd2bed999d\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " Aug 04 19:11:31.951578 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.951526 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c663496f-8313-4a90-854d-b5dd2bed999d-istio-podinfo\") pod \"c663496f-8313-4a90-854d-b5dd2bed999d\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " Aug 04 19:11:31.951578 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.951554 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c663496f-8313-4a90-854d-b5dd2bed999d-istiod-ca-cert\") pod \"c663496f-8313-4a90-854d-b5dd2bed999d\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " Aug 04 19:11:31.951836 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.951593 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-credential-socket\") pod \"c663496f-8313-4a90-854d-b5dd2bed999d\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " Aug 04 19:11:31.951836 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.951619 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c663496f-8313-4a90-854d-b5dd2bed999d-istio-token\") pod \"c663496f-8313-4a90-854d-b5dd2bed999d\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " Aug 04 19:11:31.951836 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.951654 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bf5jw\" (UniqueName: \"kubernetes.io/projected/c663496f-8313-4a90-854d-b5dd2bed999d-kube-api-access-bf5jw\") pod \"c663496f-8313-4a90-854d-b5dd2bed999d\" (UID: \"c663496f-8313-4a90-854d-b5dd2bed999d\") " Aug 04 19:11:31.951836 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.951723 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c663496f-8313-4a90-854d-b5dd2bed999d" (UID: "c663496f-8313-4a90-854d-b5dd2bed999d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:11:31.951836 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.951762 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c663496f-8313-4a90-854d-b5dd2bed999d" (UID: "c663496f-8313-4a90-854d-b5dd2bed999d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:11:31.952127 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.951841 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c663496f-8313-4a90-854d-b5dd2bed999d" (UID: "c663496f-8313-4a90-854d-b5dd2bed999d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:11:31.952127 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.951920 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-istio-data" (OuterVolumeSpecName: "istio-data") pod "c663496f-8313-4a90-854d-b5dd2bed999d" (UID: "c663496f-8313-4a90-854d-b5dd2bed999d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:11:31.952127 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.952013 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c663496f-8313-4a90-854d-b5dd2bed999d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c663496f-8313-4a90-854d-b5dd2bed999d" (UID: "c663496f-8313-4a90-854d-b5dd2bed999d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:11:31.952127 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.952091 2526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-istio-data\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:11:31.952127 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.952118 2526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-credential-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:11:31.952312 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.952137 2526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-workload-certs\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:11:31.952312 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.952150 2526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-workload-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:11:31.953995 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.953948 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c663496f-8313-4a90-854d-b5dd2bed999d-kube-api-access-bf5jw" (OuterVolumeSpecName: "kube-api-access-bf5jw") pod "c663496f-8313-4a90-854d-b5dd2bed999d" (UID: "c663496f-8313-4a90-854d-b5dd2bed999d"). InnerVolumeSpecName "kube-api-access-bf5jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:11:31.953995 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.953950 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c663496f-8313-4a90-854d-b5dd2bed999d" (UID: "c663496f-8313-4a90-854d-b5dd2bed999d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:11:31.954209 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.954187 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c663496f-8313-4a90-854d-b5dd2bed999d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c663496f-8313-4a90-854d-b5dd2bed999d" (UID: "c663496f-8313-4a90-854d-b5dd2bed999d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 19:11:31.954270 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:31.954252 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c663496f-8313-4a90-854d-b5dd2bed999d-istio-token" (OuterVolumeSpecName: "istio-token") pod "c663496f-8313-4a90-854d-b5dd2bed999d" (UID: "c663496f-8313-4a90-854d-b5dd2bed999d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:11:32.052726 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:32.052687 2526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c663496f-8313-4a90-854d-b5dd2bed999d-istio-envoy\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:11:32.052726 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:32.052715 2526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c663496f-8313-4a90-854d-b5dd2bed999d-istio-podinfo\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:11:32.052726 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:32.052724 2526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c663496f-8313-4a90-854d-b5dd2bed999d-istiod-ca-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:11:32.052726 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:32.052732 2526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c663496f-8313-4a90-854d-b5dd2bed999d-istio-token\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:11:32.053021 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:32.052746 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf5jw\" (UniqueName: \"kubernetes.io/projected/c663496f-8313-4a90-854d-b5dd2bed999d-kube-api-access-bf5jw\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:11:32.799321 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:32.799279 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" Aug 04 19:11:32.799321 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:32.799312 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc" event={"ID":"c663496f-8313-4a90-854d-b5dd2bed999d","Type":"ContainerDied","Data":"f4d4d1c4ac587020360b989a156385a9f4b848f5c56409e287cacfe5de490f82"} Aug 04 19:11:32.799857 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:32.799375 2526 scope.go:117] "RemoveContainer" containerID="d45bd7cd672de2f9ca16d333d71d47c097f7b84bf6e90ba9a0360412ab417af7" Aug 04 19:11:32.828534 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:32.828495 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc"] Aug 04 19:11:32.830380 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:32.830358 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0c0dc7-openshift-default-55b94554d-vs6kc"] Aug 04 19:11:33.181189 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:33.181103 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c663496f-8313-4a90-854d-b5dd2bed999d" path="/var/lib/kubelet/pods/c663496f-8313-4a90-854d-b5dd2bed999d/volumes" Aug 04 19:11:34.109732 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.109694 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f"] Aug 04 19:11:34.110224 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.110005 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" podUID="8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c" containerName="maas-api" containerID="cri-o://0551d84ebb8d3f202b112dd34711b7cc4d2b030b73d3b2f9a3d050c213afbae8" gracePeriod=30 Aug 04 19:11:34.161487 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.161433 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:34.208029 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.207999 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk"] Aug 04 19:11:34.208241 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.208205 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" podUID="3290241b-e9b7-4fce-b3eb-bc729b3cf49b" containerName="payload-pre-processing" containerID="cri-o://747815af96ce072baa7fbd5998ec5cabd1f34722e5353440ab3c3d2d72d5a451" gracePeriod=30 Aug 04 19:11:34.279456 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.279421 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:34.386203 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.386178 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" Aug 04 19:11:34.408247 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.408204 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:34.476337 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.476312 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwkf9\" (UniqueName: \"kubernetes.io/projected/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-kube-api-access-dwkf9\") pod \"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c\" (UID: \"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c\") " Aug 04 19:11:34.476468 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.476434 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-openshift-service-ca\") pod \"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c\" (UID: \"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c\") " Aug 04 19:11:34.476533 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.476488 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-maas-api-tls\") pod \"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c\" (UID: \"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c\") " Aug 04 19:11:34.476878 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.476842 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c" (UID: "8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:11:34.478593 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.478571 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c" (UID: "8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:11:34.478807 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.478792 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-kube-api-access-dwkf9" (OuterVolumeSpecName: "kube-api-access-dwkf9") pod "8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c" (UID: "8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c"). InnerVolumeSpecName "kube-api-access-dwkf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:11:34.486502 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.486478 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" Aug 04 19:11:34.555202 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.555169 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:34.578072 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.578043 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdglz\" (UniqueName: \"kubernetes.io/projected/3290241b-e9b7-4fce-b3eb-bc729b3cf49b-kube-api-access-vdglz\") pod \"3290241b-e9b7-4fce-b3eb-bc729b3cf49b\" (UID: \"3290241b-e9b7-4fce-b3eb-bc729b3cf49b\") " Aug 04 19:11:34.578238 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.578082 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3290241b-e9b7-4fce-b3eb-bc729b3cf49b-plugins-config-volume\") pod \"3290241b-e9b7-4fce-b3eb-bc729b3cf49b\" (UID: \"3290241b-e9b7-4fce-b3eb-bc729b3cf49b\") " Aug 04 19:11:34.578302 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.578276 2526 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-openshift-service-ca\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:11:34.578302 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.578288 2526 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-maas-api-tls\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:11:34.578302 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.578299 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwkf9\" (UniqueName: \"kubernetes.io/projected/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c-kube-api-access-dwkf9\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:11:34.578472 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.578448 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3290241b-e9b7-4fce-b3eb-bc729b3cf49b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3290241b-e9b7-4fce-b3eb-bc729b3cf49b" (UID: "3290241b-e9b7-4fce-b3eb-bc729b3cf49b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:11:34.580145 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.580123 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3290241b-e9b7-4fce-b3eb-bc729b3cf49b-kube-api-access-vdglz" (OuterVolumeSpecName: "kube-api-access-vdglz") pod "3290241b-e9b7-4fce-b3eb-bc729b3cf49b" (UID: "3290241b-e9b7-4fce-b3eb-bc729b3cf49b"). InnerVolumeSpecName "kube-api-access-vdglz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:11:34.679771 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.679688 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdglz\" (UniqueName: \"kubernetes.io/projected/3290241b-e9b7-4fce-b3eb-bc729b3cf49b-kube-api-access-vdglz\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:11:34.679771 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.679717 2526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3290241b-e9b7-4fce-b3eb-bc729b3cf49b-plugins-config-volume\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:11:34.812608 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.812575 2526 generic.go:356] "Generic (PLEG): container finished" podID="8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c" containerID="0551d84ebb8d3f202b112dd34711b7cc4d2b030b73d3b2f9a3d050c213afbae8" exitCode=0 Aug 04 19:11:34.812796 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.812622 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" Aug 04 19:11:34.812796 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.812661 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" event={"ID":"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c","Type":"ContainerDied","Data":"0551d84ebb8d3f202b112dd34711b7cc4d2b030b73d3b2f9a3d050c213afbae8"} Aug 04 19:11:34.812796 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.812727 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f" event={"ID":"8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c","Type":"ContainerDied","Data":"47c9a7ed0d9b5fd799f503af9b0118c35a123282b13a63e5316c9f1b42c41ca9"} Aug 04 19:11:34.812796 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.812758 2526 scope.go:117] "RemoveContainer" containerID="0551d84ebb8d3f202b112dd34711b7cc4d2b030b73d3b2f9a3d050c213afbae8" Aug 04 19:11:34.814209 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.814190 2526 generic.go:356] "Generic (PLEG): container finished" podID="3290241b-e9b7-4fce-b3eb-bc729b3cf49b" containerID="747815af96ce072baa7fbd5998ec5cabd1f34722e5353440ab3c3d2d72d5a451" exitCode=0 Aug 04 19:11:34.814315 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.814244 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" event={"ID":"3290241b-e9b7-4fce-b3eb-bc729b3cf49b","Type":"ContainerDied","Data":"747815af96ce072baa7fbd5998ec5cabd1f34722e5353440ab3c3d2d72d5a451"} Aug 04 19:11:34.814315 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.814274 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" event={"ID":"3290241b-e9b7-4fce-b3eb-bc729b3cf49b","Type":"ContainerDied","Data":"d6ef908c3394ba301c573f8052198e258916ac225788450c276dfcb5aa111afe"} Aug 04 19:11:34.814315 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.814273 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk" Aug 04 19:11:34.823250 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.823231 2526 scope.go:117] "RemoveContainer" containerID="0551d84ebb8d3f202b112dd34711b7cc4d2b030b73d3b2f9a3d050c213afbae8" Aug 04 19:11:34.823486 ip-10-0-136-165 kubenswrapper[2526]: E0804 19:11:34.823469 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0551d84ebb8d3f202b112dd34711b7cc4d2b030b73d3b2f9a3d050c213afbae8\": container with ID starting with 0551d84ebb8d3f202b112dd34711b7cc4d2b030b73d3b2f9a3d050c213afbae8 not found: ID does not exist" containerID="0551d84ebb8d3f202b112dd34711b7cc4d2b030b73d3b2f9a3d050c213afbae8" Aug 04 19:11:34.823542 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.823495 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0551d84ebb8d3f202b112dd34711b7cc4d2b030b73d3b2f9a3d050c213afbae8"} err="failed to get container status \"0551d84ebb8d3f202b112dd34711b7cc4d2b030b73d3b2f9a3d050c213afbae8\": rpc error: code = NotFound desc = could not find container \"0551d84ebb8d3f202b112dd34711b7cc4d2b030b73d3b2f9a3d050c213afbae8\": container with ID starting with 0551d84ebb8d3f202b112dd34711b7cc4d2b030b73d3b2f9a3d050c213afbae8 not found: ID does not exist" Aug 04 19:11:34.823542 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.823513 2526 scope.go:117] "RemoveContainer" containerID="747815af96ce072baa7fbd5998ec5cabd1f34722e5353440ab3c3d2d72d5a451" Aug 04 19:11:34.831342 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.831324 2526 scope.go:117] "RemoveContainer" containerID="747815af96ce072baa7fbd5998ec5cabd1f34722e5353440ab3c3d2d72d5a451" Aug 04 19:11:34.831607 ip-10-0-136-165 kubenswrapper[2526]: E0804 19:11:34.831589 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747815af96ce072baa7fbd5998ec5cabd1f34722e5353440ab3c3d2d72d5a451\": container with ID starting with 747815af96ce072baa7fbd5998ec5cabd1f34722e5353440ab3c3d2d72d5a451 not found: ID does not exist" containerID="747815af96ce072baa7fbd5998ec5cabd1f34722e5353440ab3c3d2d72d5a451" Aug 04 19:11:34.831656 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.831614 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747815af96ce072baa7fbd5998ec5cabd1f34722e5353440ab3c3d2d72d5a451"} err="failed to get container status \"747815af96ce072baa7fbd5998ec5cabd1f34722e5353440ab3c3d2d72d5a451\": rpc error: code = NotFound desc = could not find container \"747815af96ce072baa7fbd5998ec5cabd1f34722e5353440ab3c3d2d72d5a451\": container with ID starting with 747815af96ce072baa7fbd5998ec5cabd1f34722e5353440ab3c3d2d72d5a451 not found: ID does not exist" Aug 04 19:11:34.856934 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.856905 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f"] Aug 04 19:11:34.862884 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.862864 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4b6c58-69666fdbd4-l887f"] Aug 04 19:11:34.878046 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.878024 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk"] Aug 04 19:11:34.882995 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:34.882975 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4b6c58-6b7d798588-cm7lk"] Aug 04 19:11:35.129001 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:35.128949 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:35.180875 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:35.180845 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3290241b-e9b7-4fce-b3eb-bc729b3cf49b" path="/var/lib/kubelet/pods/3290241b-e9b7-4fce-b3eb-bc729b3cf49b/volumes" Aug 04 19:11:35.181292 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:35.181278 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c" path="/var/lib/kubelet/pods/8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c/volumes" Aug 04 19:11:42.871773 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:42.871735 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:43.000600 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:43.000561 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:43.135079 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:43.134990 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:43.244554 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:43.244522 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:50.447654 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:50.447615 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:50.549139 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:50.549107 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:50.643533 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:50.643505 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:50.739688 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:50.739655 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:51.258204 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:51.258170 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:58.562546 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:58.562487 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:58.784985 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:58.784925 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:58.883431 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:58.883347 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj"] Aug 04 19:11:58.883605 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:58.883563 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" podUID="de0b1d7c-d787-453d-9656-949d489b2c13" containerName="istio-proxy" containerID="cri-o://dd16035de02ea65a985c7dfef8950cb9ec162761f7bf3ffe49a8447bec905eca" gracePeriod=30 Aug 04 19:11:58.913699 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:58.913670 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:11:59.133910 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:11:59.133822 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:12:04.144046 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.144023 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 19:12:04.263515 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.263487 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0b1d7c-d787-453d-9656-949d489b2c13-istiod-ca-cert\") pod \"de0b1d7c-d787-453d-9656-949d489b2c13\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " Aug 04 19:12:04.263688 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.263534 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-istio-data\") pod \"de0b1d7c-d787-453d-9656-949d489b2c13\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " Aug 04 19:12:04.263688 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.263569 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-istio-envoy\") pod \"de0b1d7c-d787-453d-9656-949d489b2c13\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " Aug 04 19:12:04.263688 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.263610 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj7t8\" (UniqueName: \"kubernetes.io/projected/de0b1d7c-d787-453d-9656-949d489b2c13-kube-api-access-tj7t8\") pod \"de0b1d7c-d787-453d-9656-949d489b2c13\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " Aug 04 19:12:04.263688 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.263640 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0b1d7c-d787-453d-9656-949d489b2c13-istio-podinfo\") pod \"de0b1d7c-d787-453d-9656-949d489b2c13\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " Aug 04 19:12:04.263688 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.263666 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-credential-socket\") pod \"de0b1d7c-d787-453d-9656-949d489b2c13\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " Aug 04 19:12:04.263953 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.263866 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-workload-certs\") pod \"de0b1d7c-d787-453d-9656-949d489b2c13\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " Aug 04 19:12:04.263953 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.263887 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-istio-data" (OuterVolumeSpecName: "istio-data") pod "de0b1d7c-d787-453d-9656-949d489b2c13" (UID: "de0b1d7c-d787-453d-9656-949d489b2c13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:12:04.263953 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.263922 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-workload-socket\") pod \"de0b1d7c-d787-453d-9656-949d489b2c13\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " Aug 04 19:12:04.264150 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.263978 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0b1d7c-d787-453d-9656-949d489b2c13-istio-token\") pod \"de0b1d7c-d787-453d-9656-949d489b2c13\" (UID: \"de0b1d7c-d787-453d-9656-949d489b2c13\") " Aug 04 19:12:04.264150 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.263952 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0b1d7c-d787-453d-9656-949d489b2c13-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "de0b1d7c-d787-453d-9656-949d489b2c13" (UID: "de0b1d7c-d787-453d-9656-949d489b2c13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:12:04.264150 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.263998 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "de0b1d7c-d787-453d-9656-949d489b2c13" (UID: "de0b1d7c-d787-453d-9656-949d489b2c13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:12:04.264150 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.264129 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "de0b1d7c-d787-453d-9656-949d489b2c13" (UID: "de0b1d7c-d787-453d-9656-949d489b2c13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:12:04.264343 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.264209 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "de0b1d7c-d787-453d-9656-949d489b2c13" (UID: "de0b1d7c-d787-453d-9656-949d489b2c13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:12:04.264417 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.264368 2526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-istio-data\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:04.264417 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.264385 2526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-credential-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:04.264417 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.264399 2526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-workload-certs\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:04.264417 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.264412 2526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-workload-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:04.264618 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.264425 2526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0b1d7c-d787-453d-9656-949d489b2c13-istiod-ca-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:04.266149 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.266120 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de0b1d7c-d787-453d-9656-949d489b2c13-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "de0b1d7c-d787-453d-9656-949d489b2c13" (UID: "de0b1d7c-d787-453d-9656-949d489b2c13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 19:12:04.266273 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.266145 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "de0b1d7c-d787-453d-9656-949d489b2c13" (UID: "de0b1d7c-d787-453d-9656-949d489b2c13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:12:04.266273 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.266127 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0b1d7c-d787-453d-9656-949d489b2c13-istio-token" (OuterVolumeSpecName: "istio-token") pod "de0b1d7c-d787-453d-9656-949d489b2c13" (UID: "de0b1d7c-d787-453d-9656-949d489b2c13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:12:04.266273 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.266234 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0b1d7c-d787-453d-9656-949d489b2c13-kube-api-access-tj7t8" (OuterVolumeSpecName: "kube-api-access-tj7t8") pod "de0b1d7c-d787-453d-9656-949d489b2c13" (UID: "de0b1d7c-d787-453d-9656-949d489b2c13"). InnerVolumeSpecName "kube-api-access-tj7t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:12:04.365539 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.365514 2526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0b1d7c-d787-453d-9656-949d489b2c13-istio-envoy\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:04.365539 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.365535 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj7t8\" (UniqueName: \"kubernetes.io/projected/de0b1d7c-d787-453d-9656-949d489b2c13-kube-api-access-tj7t8\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:04.365684 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.365547 2526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0b1d7c-d787-453d-9656-949d489b2c13-istio-podinfo\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:04.365684 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.365558 2526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0b1d7c-d787-453d-9656-949d489b2c13-istio-token\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:04.901348 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.901315 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:12:04.969925 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.969844 2526 generic.go:356] "Generic (PLEG): container finished" podID="de0b1d7c-d787-453d-9656-949d489b2c13" containerID="dd16035de02ea65a985c7dfef8950cb9ec162761f7bf3ffe49a8447bec905eca" exitCode=0 Aug 04 19:12:04.970134 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.969921 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" Aug 04 19:12:04.970134 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.970076 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" event={"ID":"de0b1d7c-d787-453d-9656-949d489b2c13","Type":"ContainerDied","Data":"dd16035de02ea65a985c7dfef8950cb9ec162761f7bf3ffe49a8447bec905eca"} Aug 04 19:12:04.970134 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.970127 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj" event={"ID":"de0b1d7c-d787-453d-9656-949d489b2c13","Type":"ContainerDied","Data":"40750a107f8e4321be87c108fb9283a73a1268109a2d18e2914b6048866e9d47"} Aug 04 19:12:04.970265 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.970158 2526 scope.go:117] "RemoveContainer" containerID="dd16035de02ea65a985c7dfef8950cb9ec162761f7bf3ffe49a8447bec905eca" Aug 04 19:12:04.980786 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.980764 2526 scope.go:117] "RemoveContainer" containerID="5435b7d7b8a49a9d51589106a5e8f5b7013a56554e8649b26dd8b4f1d68f207b" Aug 04 19:12:04.989023 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.989009 2526 scope.go:117] "RemoveContainer" containerID="dd16035de02ea65a985c7dfef8950cb9ec162761f7bf3ffe49a8447bec905eca" Aug 04 19:12:04.989277 ip-10-0-136-165 kubenswrapper[2526]: E0804 19:12:04.989261 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd16035de02ea65a985c7dfef8950cb9ec162761f7bf3ffe49a8447bec905eca\": container with ID starting with dd16035de02ea65a985c7dfef8950cb9ec162761f7bf3ffe49a8447bec905eca not found: ID does not exist" containerID="dd16035de02ea65a985c7dfef8950cb9ec162761f7bf3ffe49a8447bec905eca" Aug 04 19:12:04.989338 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.989285 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd16035de02ea65a985c7dfef8950cb9ec162761f7bf3ffe49a8447bec905eca"} err="failed to get container status \"dd16035de02ea65a985c7dfef8950cb9ec162761f7bf3ffe49a8447bec905eca\": rpc error: code = NotFound desc = could not find container \"dd16035de02ea65a985c7dfef8950cb9ec162761f7bf3ffe49a8447bec905eca\": container with ID starting with dd16035de02ea65a985c7dfef8950cb9ec162761f7bf3ffe49a8447bec905eca not found: ID does not exist" Aug 04 19:12:04.989338 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.989302 2526 scope.go:117] "RemoveContainer" containerID="5435b7d7b8a49a9d51589106a5e8f5b7013a56554e8649b26dd8b4f1d68f207b" Aug 04 19:12:04.989525 ip-10-0-136-165 kubenswrapper[2526]: E0804 19:12:04.989508 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5435b7d7b8a49a9d51589106a5e8f5b7013a56554e8649b26dd8b4f1d68f207b\": container with ID starting with 5435b7d7b8a49a9d51589106a5e8f5b7013a56554e8649b26dd8b4f1d68f207b not found: ID does not exist" containerID="5435b7d7b8a49a9d51589106a5e8f5b7013a56554e8649b26dd8b4f1d68f207b" Aug 04 19:12:04.989582 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:04.989535 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5435b7d7b8a49a9d51589106a5e8f5b7013a56554e8649b26dd8b4f1d68f207b"} err="failed to get container status \"5435b7d7b8a49a9d51589106a5e8f5b7013a56554e8649b26dd8b4f1d68f207b\": rpc error: code = NotFound desc = could not find container \"5435b7d7b8a49a9d51589106a5e8f5b7013a56554e8649b26dd8b4f1d68f207b\": container with ID starting with 5435b7d7b8a49a9d51589106a5e8f5b7013a56554e8649b26dd8b4f1d68f207b not found: ID does not exist" Aug 04 19:12:05.000420 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:05.000393 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj"] Aug 04 19:12:05.004219 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:05.004199 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-219d48-openshift-default-64d4bfb9bd-7g2hj"] Aug 04 19:12:05.090096 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:05.090068 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:12:05.181569 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:05.181488 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de0b1d7c-d787-453d-9656-949d489b2c13" path="/var/lib/kubelet/pods/de0b1d7c-d787-453d-9656-949d489b2c13/volumes" Aug 04 19:12:05.192958 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:05.192930 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:12:05.296867 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:05.296834 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:12:05.681127 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:05.681094 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:12:12.092662 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:12.092631 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:12:12.205786 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:12.205745 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:12:12.363839 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:12.363754 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s"] Aug 04 19:12:12.364058 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:12.364026 2526 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" podUID="fd77988c-6b5c-4a10-af94-9da49d0446ca" containerName="istio-proxy" containerID="cri-o://e80c1a3c8a29e377cb39531b4db4863f6cb791994c8dc367f92c2ed878767d22" gracePeriod=30 Aug 04 19:12:12.429634 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:12.429603 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:12:12.737101 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:12.737073 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:12:13.847568 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:13.847533 2526 patch_prober.go:28] interesting pod/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 19:12:13.847988 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:13.847609 2526 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" podUID="fd77988c-6b5c-4a10-af94-9da49d0446ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 19:12:17.611311 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.611287 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 19:12:17.681228 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.681168 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-workload-socket\") pod \"fd77988c-6b5c-4a10-af94-9da49d0446ca\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " Aug 04 19:12:17.681228 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.681218 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-workload-certs\") pod \"fd77988c-6b5c-4a10-af94-9da49d0446ca\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " Aug 04 19:12:17.681416 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.681251 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-podinfo\") pod \"fd77988c-6b5c-4a10-af94-9da49d0446ca\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " Aug 04 19:12:17.681416 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.681282 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b65lv\" (UniqueName: \"kubernetes.io/projected/fd77988c-6b5c-4a10-af94-9da49d0446ca-kube-api-access-b65lv\") pod \"fd77988c-6b5c-4a10-af94-9da49d0446ca\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " Aug 04 19:12:17.681416 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.681373 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fd77988c-6b5c-4a10-af94-9da49d0446ca" (UID: "fd77988c-6b5c-4a10-af94-9da49d0446ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:12:17.681865 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.681467 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fd77988c-6b5c-4a10-af94-9da49d0446ca" (UID: "fd77988c-6b5c-4a10-af94-9da49d0446ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:12:17.681865 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.681474 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd77988c-6b5c-4a10-af94-9da49d0446ca-istiod-ca-cert\") pod \"fd77988c-6b5c-4a10-af94-9da49d0446ca\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " Aug 04 19:12:17.681865 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.681508 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-token\") pod \"fd77988c-6b5c-4a10-af94-9da49d0446ca\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " Aug 04 19:12:17.681865 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.681539 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-envoy\") pod \"fd77988c-6b5c-4a10-af94-9da49d0446ca\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " Aug 04 19:12:17.681865 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.681565 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-data\") pod \"fd77988c-6b5c-4a10-af94-9da49d0446ca\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " Aug 04 19:12:17.681865 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.681619 2526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-credential-socket\") pod \"fd77988c-6b5c-4a10-af94-9da49d0446ca\" (UID: \"fd77988c-6b5c-4a10-af94-9da49d0446ca\") " Aug 04 19:12:17.681865 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.681808 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd77988c-6b5c-4a10-af94-9da49d0446ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fd77988c-6b5c-4a10-af94-9da49d0446ca" (UID: "fd77988c-6b5c-4a10-af94-9da49d0446ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:12:17.682273 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.681912 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "fd77988c-6b5c-4a10-af94-9da49d0446ca" (UID: "fd77988c-6b5c-4a10-af94-9da49d0446ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:12:17.682273 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.682011 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fd77988c-6b5c-4a10-af94-9da49d0446ca" (UID: "fd77988c-6b5c-4a10-af94-9da49d0446ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:12:17.682273 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.682064 2526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-workload-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:17.682273 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.682075 2526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-workload-certs\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:17.682273 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.682086 2526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd77988c-6b5c-4a10-af94-9da49d0446ca-istiod-ca-cert\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:17.683482 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.683459 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd77988c-6b5c-4a10-af94-9da49d0446ca-kube-api-access-b65lv" (OuterVolumeSpecName: "kube-api-access-b65lv") pod "fd77988c-6b5c-4a10-af94-9da49d0446ca" (UID: "fd77988c-6b5c-4a10-af94-9da49d0446ca"). InnerVolumeSpecName "kube-api-access-b65lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:12:17.683608 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.683530 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fd77988c-6b5c-4a10-af94-9da49d0446ca" (UID: "fd77988c-6b5c-4a10-af94-9da49d0446ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 19:12:17.683906 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.683875 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fd77988c-6b5c-4a10-af94-9da49d0446ca" (UID: "fd77988c-6b5c-4a10-af94-9da49d0446ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:12:17.684021 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.683881 2526 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd77988c-6b5c-4a10-af94-9da49d0446ca" (UID: "fd77988c-6b5c-4a10-af94-9da49d0446ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:12:17.783460 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.783434 2526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-token\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:17.783460 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.783455 2526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-envoy\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:17.783460 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.783464 2526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-data\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:17.783646 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.783475 2526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd77988c-6b5c-4a10-af94-9da49d0446ca-credential-socket\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:17.783646 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.783487 2526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd77988c-6b5c-4a10-af94-9da49d0446ca-istio-podinfo\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:17.783646 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:17.783497 2526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b65lv\" (UniqueName: \"kubernetes.io/projected/fd77988c-6b5c-4a10-af94-9da49d0446ca-kube-api-access-b65lv\") on node \"ip-10-0-136-165.ec2.internal\" DevicePath \"\"" Aug 04 19:12:18.035090 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:18.035056 2526 generic.go:356] "Generic (PLEG): container finished" podID="fd77988c-6b5c-4a10-af94-9da49d0446ca" containerID="e80c1a3c8a29e377cb39531b4db4863f6cb791994c8dc367f92c2ed878767d22" exitCode=0 Aug 04 19:12:18.035259 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:18.035101 2526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" Aug 04 19:12:18.035259 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:18.035107 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" event={"ID":"fd77988c-6b5c-4a10-af94-9da49d0446ca","Type":"ContainerDied","Data":"e80c1a3c8a29e377cb39531b4db4863f6cb791994c8dc367f92c2ed878767d22"} Aug 04 19:12:18.035259 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:18.035149 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s" event={"ID":"fd77988c-6b5c-4a10-af94-9da49d0446ca","Type":"ContainerDied","Data":"ceb13ab9d1a76f0737fd4cadeaf0cd10efc7587a684c7af36868d543363c5d6a"} Aug 04 19:12:18.035259 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:18.035170 2526 scope.go:117] "RemoveContainer" containerID="e80c1a3c8a29e377cb39531b4db4863f6cb791994c8dc367f92c2ed878767d22" Aug 04 19:12:18.044592 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:18.044578 2526 scope.go:117] "RemoveContainer" containerID="91a24d907dc6a988efa2f2d0006e4d955f775a5aa4d879f8e48e8d3303cc9d82" Aug 04 19:12:18.052425 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:18.052408 2526 scope.go:117] "RemoveContainer" containerID="e80c1a3c8a29e377cb39531b4db4863f6cb791994c8dc367f92c2ed878767d22" Aug 04 19:12:18.052671 ip-10-0-136-165 kubenswrapper[2526]: E0804 19:12:18.052653 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80c1a3c8a29e377cb39531b4db4863f6cb791994c8dc367f92c2ed878767d22\": container with ID starting with e80c1a3c8a29e377cb39531b4db4863f6cb791994c8dc367f92c2ed878767d22 not found: ID does not exist" containerID="e80c1a3c8a29e377cb39531b4db4863f6cb791994c8dc367f92c2ed878767d22" Aug 04 19:12:18.052719 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:18.052678 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80c1a3c8a29e377cb39531b4db4863f6cb791994c8dc367f92c2ed878767d22"} err="failed to get container status \"e80c1a3c8a29e377cb39531b4db4863f6cb791994c8dc367f92c2ed878767d22\": rpc error: code = NotFound desc = could not find container \"e80c1a3c8a29e377cb39531b4db4863f6cb791994c8dc367f92c2ed878767d22\": container with ID starting with e80c1a3c8a29e377cb39531b4db4863f6cb791994c8dc367f92c2ed878767d22 not found: ID does not exist" Aug 04 19:12:18.052719 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:18.052698 2526 scope.go:117] "RemoveContainer" containerID="91a24d907dc6a988efa2f2d0006e4d955f775a5aa4d879f8e48e8d3303cc9d82" Aug 04 19:12:18.052912 ip-10-0-136-165 kubenswrapper[2526]: E0804 19:12:18.052897 2526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a24d907dc6a988efa2f2d0006e4d955f775a5aa4d879f8e48e8d3303cc9d82\": container with ID starting with 91a24d907dc6a988efa2f2d0006e4d955f775a5aa4d879f8e48e8d3303cc9d82 not found: ID does not exist" containerID="91a24d907dc6a988efa2f2d0006e4d955f775a5aa4d879f8e48e8d3303cc9d82" Aug 04 19:12:18.052953 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:18.052916 2526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a24d907dc6a988efa2f2d0006e4d955f775a5aa4d879f8e48e8d3303cc9d82"} err="failed to get container status \"91a24d907dc6a988efa2f2d0006e4d955f775a5aa4d879f8e48e8d3303cc9d82\": rpc error: code = NotFound desc = could not find container \"91a24d907dc6a988efa2f2d0006e4d955f775a5aa4d879f8e48e8d3303cc9d82\": container with ID starting with 91a24d907dc6a988efa2f2d0006e4d955f775a5aa4d879f8e48e8d3303cc9d82 not found: ID does not exist" Aug 04 19:12:18.063224 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:18.063204 2526 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s"] Aug 04 19:12:18.066410 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:18.066391 2526 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-de2394-openshift-default-5bb99554bd-m5t7s"] Aug 04 19:12:19.180920 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:19.180889 2526 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd77988c-6b5c-4a10-af94-9da49d0446ca" path="/var/lib/kubelet/pods/fd77988c-6b5c-4a10-af94-9da49d0446ca/volumes" Aug 04 19:12:19.311238 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:19.311212 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-2cdvs_69af5af2-c017-4455-b51d-03fe560a578b/manager/1.log" Aug 04 19:12:19.447395 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:19.447323 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7j4tr_1caa4745-97d7-40d8-b66b-ee647e71a995/manager/1.log" Aug 04 19:12:19.579534 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:19.579502 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cffffc5dc-blwt8_a7e54ecf-6cca-42e8-a708-edc70403685f/manager/1.log" Aug 04 19:12:19.700483 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:19.700458 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-gkwpb_2960465c-dc9e-49be-a178-fbeb55d9f1d2/server/0.log" Aug 04 19:12:19.836219 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:19.836193 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fdvj6_c6cc8554-0e4a-4b65-af23-661da2fcf0f3/manager/1.log" Aug 04 19:12:20.317633 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:20.317604 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76bbc69bc6-s48rg_eb341c5b-c651-4e86-a138-f9d2b0d747cf/copy-manifests/0.log" Aug 04 19:12:20.329195 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:20.329171 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76bbc69bc6-s48rg_eb341c5b-c651-4e86-a138-f9d2b0d747cf/manager/1.log" Aug 04 19:12:20.712718 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:20.712693 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-846c96f86f-fs6jb_d45be4bd-bec4-438e-a844-4b8df8730422/maas-api/0.log" Aug 04 19:12:21.291354 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:21.291330 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-g4hgd_89c68cc3-099d-4307-8375-4bcb9a5365b3/postgres/0.log" Aug 04 19:12:22.277654 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:22.277626 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-dmvk7_23b21430-239b-4799-8119-ed05095d4ee9/manager/1.log" Aug 04 19:12:22.404501 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:22.404471 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-hl5wg_e92e0caa-ebcb-4775-92c1-f66c63d895e0/manager/1.log" Aug 04 19:12:22.924080 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:22.924048 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-mtwwk_2256ff0e-84f0-4c89-a5e8-6809a4be5154/manager/1.log" Aug 04 19:12:23.287237 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:23.287207 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv_8f3e6bd2-9d7a-4354-8775-1a0f0292891c/istio-proxy/0.log" Aug 04 19:12:23.549760 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:23.549681 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8489764dcd-7bclw_32c12d64-b08e-4ca2-af73-c51e416fdd13/kube-auth-proxy/0.log" Aug 04 19:12:23.805020 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:23.804933 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6vpxf_39fbabce-fc4c-413a-ae9d-4672ba61e1d6/istio-proxy/1.log" Aug 04 19:12:23.930736 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:23.930704 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zftxf_426116a3-0958-4362-89e3-b9900c902faa/payload-pre-processing/0.log" Aug 04 19:12:24.073490 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:24.073406 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-c9kjx_3346ba4e-c7fa-4d34-83f7-2780f3b42d33/payload-processing/0.log" Aug 04 19:12:24.684550 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:24.684523 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-jz7rt_b52ef147-aa43-4597-9a81-e080819991b3/main/0.log" Aug 04 19:12:24.808148 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:24.808119 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-wg5tr_2c6e331a-0343-4b1f-b95d-800ceba00cf7/main/0.log" Aug 04 19:12:25.177979 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:25.177948 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-8h6ks_a8d9ecd5-91f5-4114-9bb9-40afc20e9d59/main/0.log" Aug 04 19:12:31.133543 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:31.133506 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:12:33.843144 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:33.843109 2526 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" limit=1024 Aug 04 19:12:35.028662 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:35.028629 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r59lx_94ea3648-f325-45aa-b53b-f5c59bd1bd5a/global-pull-secret-syncer/0.log" Aug 04 19:12:35.090913 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:35.090885 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-845wp_df338c86-7a60-48e5-ba0a-79a483d7275c/konnectivity-agent/0.log" Aug 04 19:12:35.207726 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:35.207700 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-165.ec2.internal_a598d320185d24ca207d357f68b8dce3/haproxy/0.log" Aug 04 19:12:40.384281 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:40.384244 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-dmvk7_23b21430-239b-4799-8119-ed05095d4ee9/manager/1.log" Aug 04 19:12:40.386733 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:40.386712 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-dmvk7_23b21430-239b-4799-8119-ed05095d4ee9/manager/0.log" Aug 04 19:12:40.416295 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:40.416272 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-hl5wg_e92e0caa-ebcb-4775-92c1-f66c63d895e0/manager/1.log" Aug 04 19:12:40.416389 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:40.416334 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-hl5wg_e92e0caa-ebcb-4775-92c1-f66c63d895e0/manager/0.log" Aug 04 19:12:40.718156 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:40.718121 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-mtwwk_2256ff0e-84f0-4c89-a5e8-6809a4be5154/manager/1.log" Aug 04 19:12:40.728535 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:40.728501 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-mtwwk_2256ff0e-84f0-4c89-a5e8-6809a4be5154/manager/0.log" Aug 04 19:12:42.817256 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:42.817215 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kpskw_020e5d5d-aaab-439c-8927-65a1aeb4c9a1/node-exporter/0.log" Aug 04 19:12:42.853839 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:42.853815 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kpskw_020e5d5d-aaab-439c-8927-65a1aeb4c9a1/kube-rbac-proxy/0.log" Aug 04 19:12:42.893469 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:42.893442 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kpskw_020e5d5d-aaab-439c-8927-65a1aeb4c9a1/init-textfile/0.log" Aug 04 19:12:43.723584 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:43.723554 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75df96dc46-88sdg_b65ba86c-a92d-4db7-9052-e2a40ba8f9ac/telemeter-client/0.log" Aug 04 19:12:43.769129 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:43.769105 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75df96dc46-88sdg_b65ba86c-a92d-4db7-9052-e2a40ba8f9ac/reload/0.log" Aug 04 19:12:43.798199 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:43.798158 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75df96dc46-88sdg_b65ba86c-a92d-4db7-9052-e2a40ba8f9ac/kube-rbac-proxy/0.log" Aug 04 19:12:45.241724 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.241691 2526 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq"] Aug 04 19:12:45.242149 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242082 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3290241b-e9b7-4fce-b3eb-bc729b3cf49b" containerName="payload-pre-processing" Aug 04 19:12:45.242149 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242094 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="3290241b-e9b7-4fce-b3eb-bc729b3cf49b" containerName="payload-pre-processing" Aug 04 19:12:45.242149 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242109 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0b1d7c-d787-453d-9656-949d489b2c13" containerName="istio-proxy" Aug 04 19:12:45.242149 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242114 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0b1d7c-d787-453d-9656-949d489b2c13" containerName="istio-proxy" Aug 04 19:12:45.242149 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242126 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd77988c-6b5c-4a10-af94-9da49d0446ca" containerName="istio-proxy" Aug 04 19:12:45.242149 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242136 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd77988c-6b5c-4a10-af94-9da49d0446ca" containerName="istio-proxy" Aug 04 19:12:45.242149 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242147 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c663496f-8313-4a90-854d-b5dd2bed999d" containerName="istio-proxy" Aug 04 19:12:45.242149 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242152 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="c663496f-8313-4a90-854d-b5dd2bed999d" containerName="istio-proxy" Aug 04 19:12:45.242400 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242158 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c" containerName="maas-api" Aug 04 19:12:45.242400 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242162 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c" containerName="maas-api" Aug 04 19:12:45.242400 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242169 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="445d2a74-f641-4fd9-8c11-6808510ef1e2" containerName="maas-api" Aug 04 19:12:45.242400 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242175 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="445d2a74-f641-4fd9-8c11-6808510ef1e2" containerName="maas-api" Aug 04 19:12:45.242400 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242182 2526 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c663496f-8313-4a90-854d-b5dd2bed999d" containerName="istio-proxy" Aug 04 19:12:45.242400 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242187 2526 state_mem.go:107] "Deleted CPUSet assignment" podUID="c663496f-8313-4a90-854d-b5dd2bed999d" containerName="istio-proxy" Aug 04 19:12:45.242400 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242251 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="de0b1d7c-d787-453d-9656-949d489b2c13" containerName="istio-proxy" Aug 04 19:12:45.242400 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242263 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd77988c-6b5c-4a10-af94-9da49d0446ca" containerName="istio-proxy" Aug 04 19:12:45.242400 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242268 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="c663496f-8313-4a90-854d-b5dd2bed999d" containerName="istio-proxy" Aug 04 19:12:45.242400 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242273 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="445d2a74-f641-4fd9-8c11-6808510ef1e2" containerName="maas-api" Aug 04 19:12:45.242400 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242280 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d1868f7-bb42-4cf1-8f43-0d7cc5ff302c" containerName="maas-api" Aug 04 19:12:45.242400 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.242290 2526 memory_manager.go:368] "RemoveStaleState removing state" podUID="3290241b-e9b7-4fce-b3eb-bc729b3cf49b" containerName="payload-pre-processing" Aug 04 19:12:45.246752 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.246733 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.250189 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.250103 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kwzfh\"/\"kube-root-ca.crt\"" Aug 04 19:12:45.251018 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.251003 2526 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kwzfh\"/\"default-dockercfg-gz4dk\"" Aug 04 19:12:45.251567 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.251554 2526 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kwzfh\"/\"openshift-service-ca.crt\"" Aug 04 19:12:45.255438 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.255416 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq"] Aug 04 19:12:45.321131 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.321101 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68853e09-c79b-4a99-8911-19e3cad4b209-lib-modules\") pod \"perf-node-gather-daemonset-bd4bq\" (UID: \"68853e09-c79b-4a99-8911-19e3cad4b209\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.321298 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.321145 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68853e09-c79b-4a99-8911-19e3cad4b209-podres\") pod \"perf-node-gather-daemonset-bd4bq\" (UID: \"68853e09-c79b-4a99-8911-19e3cad4b209\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.321298 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.321191 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s92zl\" (UniqueName: \"kubernetes.io/projected/68853e09-c79b-4a99-8911-19e3cad4b209-kube-api-access-s92zl\") pod \"perf-node-gather-daemonset-bd4bq\" (UID: \"68853e09-c79b-4a99-8911-19e3cad4b209\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.321298 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.321258 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68853e09-c79b-4a99-8911-19e3cad4b209-sys\") pod \"perf-node-gather-daemonset-bd4bq\" (UID: \"68853e09-c79b-4a99-8911-19e3cad4b209\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.321298 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.321298 2526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68853e09-c79b-4a99-8911-19e3cad4b209-proc\") pod \"perf-node-gather-daemonset-bd4bq\" (UID: \"68853e09-c79b-4a99-8911-19e3cad4b209\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.422277 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.422225 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68853e09-c79b-4a99-8911-19e3cad4b209-lib-modules\") pod \"perf-node-gather-daemonset-bd4bq\" (UID: \"68853e09-c79b-4a99-8911-19e3cad4b209\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.422277 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.422295 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68853e09-c79b-4a99-8911-19e3cad4b209-podres\") pod \"perf-node-gather-daemonset-bd4bq\" (UID: \"68853e09-c79b-4a99-8911-19e3cad4b209\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.422531 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.422321 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s92zl\" (UniqueName: \"kubernetes.io/projected/68853e09-c79b-4a99-8911-19e3cad4b209-kube-api-access-s92zl\") pod \"perf-node-gather-daemonset-bd4bq\" (UID: \"68853e09-c79b-4a99-8911-19e3cad4b209\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.422531 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.422342 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68853e09-c79b-4a99-8911-19e3cad4b209-sys\") pod \"perf-node-gather-daemonset-bd4bq\" (UID: \"68853e09-c79b-4a99-8911-19e3cad4b209\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.422531 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.422415 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68853e09-c79b-4a99-8911-19e3cad4b209-sys\") pod \"perf-node-gather-daemonset-bd4bq\" (UID: \"68853e09-c79b-4a99-8911-19e3cad4b209\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.422531 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.422432 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68853e09-c79b-4a99-8911-19e3cad4b209-podres\") pod \"perf-node-gather-daemonset-bd4bq\" (UID: \"68853e09-c79b-4a99-8911-19e3cad4b209\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.422531 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.422431 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68853e09-c79b-4a99-8911-19e3cad4b209-lib-modules\") pod \"perf-node-gather-daemonset-bd4bq\" (UID: \"68853e09-c79b-4a99-8911-19e3cad4b209\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.422531 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.422439 2526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68853e09-c79b-4a99-8911-19e3cad4b209-proc\") pod \"perf-node-gather-daemonset-bd4bq\" (UID: \"68853e09-c79b-4a99-8911-19e3cad4b209\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.422531 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.422491 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68853e09-c79b-4a99-8911-19e3cad4b209-proc\") pod \"perf-node-gather-daemonset-bd4bq\" (UID: \"68853e09-c79b-4a99-8911-19e3cad4b209\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.431934 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.431910 2526 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s92zl\" (UniqueName: \"kubernetes.io/projected/68853e09-c79b-4a99-8911-19e3cad4b209-kube-api-access-s92zl\") pod \"perf-node-gather-daemonset-bd4bq\" (UID: \"68853e09-c79b-4a99-8911-19e3cad4b209\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.557401 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.557304 2526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:45.690388 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:45.690360 2526 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq"] Aug 04 19:12:45.691871 ip-10-0-136-165 kubenswrapper[2526]: W0804 19:12:45.691846 2526 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod68853e09_c79b_4a99_8911_19e3cad4b209.slice/crio-c5636e87482b1c00df442bfc1a500ba6c409bd8d112d1aa223a72da97c755d47 WatchSource:0}: Error finding container c5636e87482b1c00df442bfc1a500ba6c409bd8d112d1aa223a72da97c755d47: Status 404 returned error can't find the container with id c5636e87482b1c00df442bfc1a500ba6c409bd8d112d1aa223a72da97c755d47 Aug 04 19:12:46.185143 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:46.185106 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" event={"ID":"68853e09-c79b-4a99-8911-19e3cad4b209","Type":"ContainerStarted","Data":"82318af142555c53484fc98b88aec48d8d42f4f8204fdf334288601dd1f4adc3"} Aug 04 19:12:46.185325 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:46.185160 2526 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" event={"ID":"68853e09-c79b-4a99-8911-19e3cad4b209","Type":"ContainerStarted","Data":"c5636e87482b1c00df442bfc1a500ba6c409bd8d112d1aa223a72da97c755d47"} Aug 04 19:12:46.185325 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:46.185188 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:46.206615 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:46.206561 2526 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" podStartSLOduration=1.206546685 podStartE2EDuration="1.206546685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:12:46.204498196 +0000 UTC m=+3723.455780745" watchObservedRunningTime="2026-08-04 19:12:46.206546685 +0000 UTC m=+3723.457829231" Aug 04 19:12:48.722812 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:48.722776 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kdk4v_7439e0a9-d4de-4b6a-ae78-48a889eff5a1/dns/0.log" Aug 04 19:12:48.749641 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:48.749615 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kdk4v_7439e0a9-d4de-4b6a-ae78-48a889eff5a1/kube-rbac-proxy/0.log" Aug 04 19:12:48.869373 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:48.869355 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rvt4h_fb9a403b-8397-4699-bde5-4a19d861d697/dns-node-resolver/0.log" Aug 04 19:12:49.543393 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:49.543366 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p9xc7_884f5581-c681-4785-a065-fa174e052347/node-ca/0.log" Aug 04 19:12:50.628914 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:50.628872 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv_8f3e6bd2-9d7a-4354-8775-1a0f0292891c/istio-proxy/0.log" Aug 04 19:12:51.014444 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:51.014419 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8489764dcd-7bclw_32c12d64-b08e-4ca2-af73-c51e416fdd13/kube-auth-proxy/0.log" Aug 04 19:12:51.081477 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:51.081447 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6vpxf_39fbabce-fc4c-413a-ae9d-4672ba61e1d6/istio-proxy/0.log" Aug 04 19:12:51.087778 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:51.087760 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6vpxf_39fbabce-fc4c-413a-ae9d-4672ba61e1d6/istio-proxy/1.log" Aug 04 19:12:51.120627 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:51.120608 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zftxf_426116a3-0958-4362-89e3-b9900c902faa/payload-pre-processing/0.log" Aug 04 19:12:51.157308 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:51.157287 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-c9kjx_3346ba4e-c7fa-4d34-83f7-2780f3b42d33/payload-processing/0.log" Aug 04 19:12:52.104652 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:52.104615 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xr2hn_c9e9f16d-731b-4afb-95be-304a641d3e4d/serve-healthcheck-canary/0.log" Aug 04 19:12:52.200669 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:52.200643 2526 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-bd4bq" Aug 04 19:12:52.846755 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:52.846725 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vzv57_83ceb079-6106-4ef2-a063-3506b4eb60a2/kube-rbac-proxy/0.log" Aug 04 19:12:52.871664 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:52.871640 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vzv57_83ceb079-6106-4ef2-a063-3506b4eb60a2/exporter/0.log" Aug 04 19:12:52.896697 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:52.896674 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vzv57_83ceb079-6106-4ef2-a063-3506b4eb60a2/extractor/0.log" Aug 04 19:12:55.206222 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:55.206200 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-2cdvs_69af5af2-c017-4455-b51d-03fe560a578b/manager/1.log" Aug 04 19:12:55.206614 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:55.206283 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-2cdvs_69af5af2-c017-4455-b51d-03fe560a578b/manager/0.log" Aug 04 19:12:55.359191 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:55.359163 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7j4tr_1caa4745-97d7-40d8-b66b-ee647e71a995/manager/0.log" Aug 04 19:12:55.389731 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:55.389703 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7j4tr_1caa4745-97d7-40d8-b66b-ee647e71a995/manager/1.log" Aug 04 19:12:55.542325 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:55.542293 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cffffc5dc-blwt8_a7e54ecf-6cca-42e8-a708-edc70403685f/manager/1.log" Aug 04 19:12:55.544338 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:55.544318 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cffffc5dc-blwt8_a7e54ecf-6cca-42e8-a708-edc70403685f/manager/0.log" Aug 04 19:12:55.566936 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:55.566915 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-gkwpb_2960465c-dc9e-49be-a178-fbeb55d9f1d2/server/0.log" Aug 04 19:12:55.633706 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:55.633604 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fdvj6_c6cc8554-0e4a-4b65-af23-661da2fcf0f3/manager/1.log" Aug 04 19:12:55.641261 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:55.641243 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fdvj6_c6cc8554-0e4a-4b65-af23-661da2fcf0f3/manager/0.log" Aug 04 19:12:56.063982 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:56.063931 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76bbc69bc6-s48rg_eb341c5b-c651-4e86-a138-f9d2b0d747cf/manager/1.log" Aug 04 19:12:56.078423 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:56.078395 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv_8f3e6bd2-9d7a-4354-8775-1a0f0292891c/istio-proxy/0.log" Aug 04 19:12:56.115575 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:56.115551 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76bbc69bc6-s48rg_eb341c5b-c651-4e86-a138-f9d2b0d747cf/manager/0.log" Aug 04 19:12:56.146722 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:56.146698 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76bbc69bc6-s48rg_eb341c5b-c651-4e86-a138-f9d2b0d747cf/copy-manifests/0.log" Aug 04 19:12:56.423716 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:56.423688 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8489764dcd-7bclw_32c12d64-b08e-4ca2-af73-c51e416fdd13/kube-auth-proxy/0.log" Aug 04 19:12:56.473916 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:56.473884 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6vpxf_39fbabce-fc4c-413a-ae9d-4672ba61e1d6/istio-proxy/0.log" Aug 04 19:12:56.480530 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:56.480446 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6vpxf_39fbabce-fc4c-413a-ae9d-4672ba61e1d6/istio-proxy/1.log" Aug 04 19:12:56.505371 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:56.505343 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zftxf_426116a3-0958-4362-89e3-b9900c902faa/payload-pre-processing/0.log" Aug 04 19:12:56.529278 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:56.529251 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-c9kjx_3346ba4e-c7fa-4d34-83f7-2780f3b42d33/payload-processing/0.log" Aug 04 19:12:57.537628 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:57.537595 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jw8gm_9a79ff6c-01b0-4585-ab7c-97af9c6e0327/openshift-lws-operator/0.log" Aug 04 19:12:58.035483 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:58.035450 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-c2kcv_8f3e6bd2-9d7a-4354-8775-1a0f0292891c/istio-proxy/0.log" Aug 04 19:12:58.370923 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:58.370899 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8489764dcd-7bclw_32c12d64-b08e-4ca2-af73-c51e416fdd13/kube-auth-proxy/0.log" Aug 04 19:12:58.423847 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:58.423813 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6vpxf_39fbabce-fc4c-413a-ae9d-4672ba61e1d6/istio-proxy/0.log" Aug 04 19:12:58.430503 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:58.430480 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6vpxf_39fbabce-fc4c-413a-ae9d-4672ba61e1d6/istio-proxy/1.log" Aug 04 19:12:58.450906 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:58.450885 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zftxf_426116a3-0958-4362-89e3-b9900c902faa/payload-pre-processing/0.log" Aug 04 19:12:58.471553 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:12:58.471533 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-c9kjx_3346ba4e-c7fa-4d34-83f7-2780f3b42d33/payload-processing/0.log" Aug 04 19:13:04.322463 ip-10-0-136-165 kubenswrapper[2526]: I0804 19:13:04.322434 2526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6m9tw_ceb0e538-c878-429c-b09b-bad693514c20/kube-multus/0.log"